big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,591 Commits 2 Branches 2 Tags 264 MiB
d7979079ea
Commit Graph

4631 Commits

Author SHA1 Message Date
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Lokesh Jain
cf268114c9 HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-02 12:24:55 -07:00
Uma Maheswara Rao G
35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan
f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Stephen O'Donnell
3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Erik Krogen
55b3a718e9 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-03-19 08:14:49 -07:00
Stephen O'Donnell
34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen
1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Wei-Chiu Chuang
6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri
945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B
f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Wei-Chiu Chuang
387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00
Wei-Chiu Chuang
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng." 
This reverts commit 5c10630ad8.
 2019-03-04 16:59:20 +09:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri
05e4ddeee6 HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena. 2019-02-28 10:03:05 -08:00
Erik Krogen
0feba4396f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-02-28 08:48:10 -08:00
Konstantin V Shvachko
a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Vinayakumar B
f4ae00c530 HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena." 2019-02-26 00:36:14 +05:30
Surendra Singh Lilhore
7d3b567194 HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi. 2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 2019-02-21 20:36:34 +05:30
Shweta Yakkali
1bea785020 HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 14:28:37 -08:00
Kitti Nanasi
14282e311b HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-19 12:04:56 -08:00
Inigo Goiri
1de25d134f HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR. 2019-02-18 15:30:38 -08:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Inigo Goiri
dde0ab55aa HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR. 2019-02-15 16:32:27 -08:00
Erik Krogen
64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5 HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena. 2019-02-14 22:13:14 +05:30
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Surendra Singh Lilhore
35d4f32b32 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 2019-02-13 23:06:46 +05:30
Vinayakumar B
7806403842 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:57:57 +05:30
Siyao Meng
5c10630ad8 HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-11 11:02:32 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Siyao Meng
6aa63452b3 HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-06 12:19:21 -08:00
Erik Krogen
d3de8e162b HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-02-06 09:54:47 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Inigo Goiri
0e79a86582 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 2019-02-04 11:59:48 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Wei-Chiu Chuang
d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00
Pranay Singh
1d523279da HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-25 09:02:57 -08:00
shwetayakkali@cloudera.com
f3e642d92b HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
 2019-01-23 15:57:39 -08:00
Kitti Nanasi
951cdd7e4c HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-23 14:40:57 -08:00
Surendra Singh Lilhore
221e308cb5 HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena. 2019-01-23 17:00:37 +05:30
Takanobu Asanuma
7d6792e5d2 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 2019-01-23 10:03:06 +09:00
Arpit Agarwal
1ff658b2ef HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 2019-01-21 20:44:37 -08:00
Inigo Goiri
f048512bb8 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 2019-01-16 10:14:22 -08:00
shwetayakkali@cloudera.com
4ab5260b7e HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
 2019-01-08 17:04:57 -08:00
Giovanni Matteo Fumarola
1a08302897 HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 2019-01-08 10:38:06 -08:00
Surendra Singh Lilhore
21fe77e3ab HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena. 2019-01-03 18:36:53 +05:30