big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,342 Commits 2 Branches 2 Tags 264 MiB
fc17ba172b
Commit Graph

4617 Commits

Author SHA1 Message Date
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri
05e4ddeee6 HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena. 2019-02-28 10:03:05 -08:00
Erik Krogen
0feba4396f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-02-28 08:48:10 -08:00
Konstantin V Shvachko
a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Vinayakumar B
f4ae00c530 HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena." 2019-02-26 00:36:14 +05:30
Surendra Singh Lilhore
7d3b567194 HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi. 2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 2019-02-21 20:36:34 +05:30
Shweta Yakkali
1bea785020 HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 14:28:37 -08:00
Kitti Nanasi
14282e311b HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-19 12:04:56 -08:00
Inigo Goiri
1de25d134f HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR. 2019-02-18 15:30:38 -08:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Inigo Goiri
dde0ab55aa HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR. 2019-02-15 16:32:27 -08:00
Erik Krogen
64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5 HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena. 2019-02-14 22:13:14 +05:30
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Surendra Singh Lilhore
35d4f32b32 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 2019-02-13 23:06:46 +05:30
Vinayakumar B
7806403842 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:57:57 +05:30
Siyao Meng
5c10630ad8 HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-11 11:02:32 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Siyao Meng
6aa63452b3 HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-06 12:19:21 -08:00
Erik Krogen
d3de8e162b HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-02-06 09:54:47 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Inigo Goiri
0e79a86582 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 2019-02-04 11:59:48 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Wei-Chiu Chuang
d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00
Pranay Singh
1d523279da HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-25 09:02:57 -08:00
shwetayakkali@cloudera.com
f3e642d92b HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
 2019-01-23 15:57:39 -08:00
Kitti Nanasi
951cdd7e4c HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-23 14:40:57 -08:00
Surendra Singh Lilhore
221e308cb5 HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena. 2019-01-23 17:00:37 +05:30
Takanobu Asanuma
7d6792e5d2 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 2019-01-23 10:03:06 +09:00
Arpit Agarwal
1ff658b2ef HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 2019-01-21 20:44:37 -08:00
Inigo Goiri
f048512bb8 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 2019-01-16 10:14:22 -08:00
shwetayakkali@cloudera.com
4ab5260b7e HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
 2019-01-08 17:04:57 -08:00
Giovanni Matteo Fumarola
1a08302897 HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 2019-01-08 10:38:06 -08:00
Surendra Singh Lilhore
21fe77e3ab HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena. 2019-01-03 18:36:53 +05:30
Konstantin V Shvachko
c9d7737431 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
c3efc299fe HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
ef3e1929dd HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2018-12-24 09:34:01 -08:00
Chao Sun
cbc2f932eb HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Chen Liang
652b257478 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
b6f20c36c2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
1c4265d7bc HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
b5b9b77707 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2018-12-24 09:34:00 -08:00
Chen Liang
a65bb97f5d HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2018-12-24 09:34:00 -08:00
Erik Krogen
b74a7dbf88 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2018-12-24 09:34:00 -08:00
Erik Krogen
25b63e8da8 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
28820fb6b1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2018-12-24 09:34:00 -08:00