big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,427 Commits 2 Branches 2 Tags 264 MiB
99f88c30cb
Commit Graph

850 Commits

Author SHA1 Message Date
Inigo Goiri
f048512bb8 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 2019-01-16 10:14:22 -08:00
Giovanni Matteo Fumarola
fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
shwetayakkali@cloudera.com
4ab5260b7e HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
 2019-01-08 17:04:57 -08:00
Konstantin V Shvachko
bd2a59e508 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2018-12-24 09:39:20 -08:00
Konstantin V Shvachko
c9d7737431 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
fa8550337d HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2018-12-24 09:34:01 -08:00
Chen Liang
b8ad6c85a5 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chen Liang
71cf66e29b HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
3bb92a1d9a HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2018-12-24 09:34:01 -08:00
Chen Liang
a3aab48df0 HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chen Liang
652b257478 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chao Sun
1a9ba9616a HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
25b63e8da8 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2018-12-24 09:34:00 -08:00
Chen Liang
d2e85b0b6d HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2018-12-24 09:34:00 -08:00
Erik Krogen
aa42fb0db7 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
f9fc01cd7f HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
25d8e39b5c HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Chen Liang
66ec3a67e8 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2018-12-24 09:34:00 -08:00
Erik Krogen
eae0a5d54a HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2018-12-24 09:34:00 -08:00
Chao Sun
64b7cf59bd HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Konstantin V Shvachko
0105d7f597 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
e880660a20 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Erik Krogen
a109f2b32f HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Erik Krogen
9bf0696c73 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
c9d73437e8 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2018-12-24 09:33:59 -08:00
Surendra Singh Lilhore
788e7473a4 HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 2018-12-17 11:24:57 +05:30
Giovanni Matteo Fumarola
f858f18554 HDFS-14088. RequestHedgingProxyProvider can throw NullPointerException when failover due to no lock on currentUsedProxy. Contributed by Yuxuan Wang. 2018-12-14 11:04:36 -08:00
Zsolt Venczel
aa12859890 HDFS-14121. Log message about the old hosts file format is misleading 
(Contributed by Zsolt Venczel via Daniel Templeton)

Change-Id: I7ff548f6c82e0aeb08a7a50ca7c2c827db8726bb
 2018-12-14 13:54:57 +01:00
Daniel Templeton
4aa0609fb0 HDFS-13985. Clearer error message for ReplicaNotFoundException 
(Contributed by Adam Antal via Daniel Templeton)

Change-Id: I68ae7a5bedecbc204c161fc3b3ce8878fe5d493d
 2018-12-13 16:21:45 +01:00
Vinayakumar B
39dc7345b8 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 2018-12-11 17:59:04 +05:30
Brahma Reddy Battula
f534736867 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 2018-11-30 00:18:27 +05:30
Brahma Reddy Battula
892b33e054 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 2018-11-22 00:13:56 +05:30
Jason Lowe
0b6625a973 MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan 2018-11-07 08:20:49 -06:00
Inigo Goiri
887244de4a HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 2018-11-06 11:18:15 -08:00
Pranay Singh
d98b881e9a HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-11-01 17:26:00 -07:00
Xiao Chen
db7e636824 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 2018-10-29 19:06:15 -07:00
Chen Liang
635786a511 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2018-10-23 14:53:45 -07:00
Inigo Goiri
88cce32551 HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin. 2018-10-19 10:38:04 -07:00
Xiao Chen
b6fc72a025 HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 2018-10-15 10:51:55 -07:00
Vinayakumar B
b60ca37914 Fix potential FSImage corruption. Contributed by Daryn Sharp. 2018-10-15 15:48:26 +05:30
Xiao Chen
5ec86b445c HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 09:35:52 -07:00
Hrishikesh Gadre
bf3d591f0c HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:23:07 -07:00
Weiwei Yang
3ead525c71 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-10 10:11:10 -07:00
Kitti Nanasi
10185d9a77 HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-10 08:55:32 -07:00
Hrishikesh Gadre
08bb6c49a5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-08 20:31:57 -07:00
Siyao Meng
396ce0d9f4 HDFS-13877. HttpFS: Implement GETSNAPSHOTDIFF. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-04 15:17:47 -07:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Takanobu Asanuma
f6c5ef9903 HDFS-13943. [JDK10] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Akira Ajisaka. 2018-10-02 09:49:48 +09:00
Wei-Chiu Chuang
28ceb34a72 HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 2018-09-19 03:12:20 -07:00
Wei-Chiu Chuang
a1de8cbac5 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-09-14 05:22:56 +08:00
Vinayakumar B
30eceec342 HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 2018-09-10 09:10:51 +05:30
Giovanni Matteo Fumarola
eca1a4bfe9 HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering. 2018-09-06 14:48:00 -07:00
Kitti Nanasi
6e5ffb74dd HDFS-13885. Add debug logs in dfsclient around decrypting EDEK. Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-03 22:37:37 -07:00
Wei-Chiu Chuang
8aa6c4f079 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 26c2a97c56.
 2018-08-30 11:44:20 -07:00
Wei-Chiu Chuang
26c2a97c56 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-08-27 16:02:35 -07:00
Konstantin V Shvachko
a4121c71c2 HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 2018-08-24 18:27:30 -07:00
Xiao Chen
c67b0650ea HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 2018-08-16 23:14:21 -07:00
Uma Maheswara Rao G
39ed3a66db HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R. 2018-08-12 03:06:06 -07:00
Uma Maheswara Rao G
4402f3f855 HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh. 2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
3b83110d5e HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R. 2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
c561cb316e HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
68017e3349 HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore. 2018-08-12 03:06:02 -07:00
Uma Maheswara Rao G
6215e35bb6 HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu 2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
cd5262aba0 HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou 2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
b67ae6d9d7 HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu 2018-08-12 03:05:54 -07:00
Xiao Chen
9499df7b81 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-08 10:40:20 -07:00
Daniel Templeton
849c45db18 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica 
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
 2018-07-24 16:05:27 -07:00
Wei-Chiu Chuang
0247cb6318 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel. 2018-07-06 14:59:49 -07:00
Andrew Wang
344f324710 HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas. 2018-07-03 11:07:45 +02:00
Inigo Goiri
1804a31515 HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-07-02 10:48:20 -07:00
Wei-Chiu Chuang
fef20a446f HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota. 2018-07-02 09:41:15 -07:00
Andrew Wang
5d748bd056 HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon. 2018-07-02 12:11:06 +02:00
Andrew Wang
6ba9974108 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon. 2018-07-02 12:02:19 +02:00
Todd Lipcon
7a3c6e9c3c HADOOP-15550. Avoid static initialization of ObjectMappers 2018-06-25 15:36:45 -07:00
Akira Ajisaka
fba9d7cd74
HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. Contributed by Takanobu Asanuma. 2018-06-18 10:17:07 -07:00
Chris Douglas
980031bb04 HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs 2018-06-17 11:54:26 -07:00
Inigo Goiri
2e5cfe6df3 HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 2018-06-11 11:20:29 -07:00
Arpit Agarwal
000a678396 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:12 -07:00
Chao Sun
fba1c42adc HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun. 2018-06-08 16:36:42 -07:00
Xiao Chen
774c1f199e HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota. 2018-06-05 21:25:08 -07:00
Xiao Chen
ba4011d64f HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel. 2018-06-05 21:18:31 -07:00
Xiao Chen
6d5e87aec2 HADOOP-15507. Add MapReduce counters about EC bytes read. 2018-06-04 21:13:47 -07:00
Wei-Chiu Chuang
c95b9b5c07 HDFS-13440. Log HDFS file name when client fails to connect. Contributed by Gabor Bota. 2018-05-31 15:20:59 -07:00
Sean Mackrory
1361030e59 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel. 2018-05-31 07:59:21 -06:00
Andrew Wang
c9b63deb53 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient. 2018-05-24 09:56:23 -07:00
Bharat Viswanadham
bc6d9d4c79 HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru 2018-05-23 10:15:40 -07:00
Sammi Chen
34e8b9f9a8 HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen. 2018-05-23 19:10:09 +08:00
Andrew Wang
1d2640b613 HDFS-13601. Optimize ByteString conversions in PBHelper. 2018-05-22 23:55:20 -07:00
Arpit Agarwal
0fc988e6a3 HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar. 2018-05-16 11:28:39 -07:00
Xiao Chen
960940e0e0 HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure. 2018-05-14 09:28:39 -07:00
Ewan Higgs
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-27 19:21:37 +09:00
Lei Xu
a8e428b2dc HDFS-13468. Add erasure coding metrics into ReadStatistics. (Contributed by Lei (Eddy) Xu) 2018-04-26 13:54:40 -07:00
Takanobu Asanuma
6f655aed3b HADOOP-15404. Remove multibyte characters in DataNodeUsageReportUtil 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-26 13:54:40 -07:00
Nanda kumar
0ec88ea42b HDFS-13301. Ozone: Remove containerPort, ratisPort and ozoneRestPort from DatanodeID and DatanodeIDProto. Contributed by Shashikant Banerjee. 2018-04-26 05:36:04 -07:00
Anu Engineer
4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects. 
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
 2018-04-26 05:36:04 -07:00
Anu Engineer
5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key. 
Contributed by Mukul Kumar Singh.
 2018-04-26 05:36:04 -07:00
Nanda kumar
9a914126a7 HDFS-13008. Ozone: Add DN container open/close state to container report. Contributed by Xiaoyu Yao. 2018-04-26 05:36:04 -07:00
Nanda kumar
b4a3cf1476 HDFS-13070. Ozone: SCM: Support for container replica reconciliation - 1. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00