big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,345 Commits 2 Branches 2 Tags 264 MiB
6c4d566955
Commit Graph

8113 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng." 
This reverts commit 5c10630ad8.
 2019-03-04 16:59:20 +09:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri
80b77deb42 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 2019-03-01 09:18:51 -08:00
Brahma Reddy Battula
7a0db2f92b HDFS-13997. Secondary NN Web UI displays nothing, and the console log shows moment is not defined. Contributed by Ayush Saxena 2019-03-01 12:24:52 +05:30
Inigo Goiri
05e4ddeee6 HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena. 2019-02-28 10:03:05 -08:00
Erik Krogen
0feba4396f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-02-28 08:48:10 -08:00
Konstantin V Shvachko
a6ab37192a HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko. 2019-02-25 14:35:02 -08:00
Vinayakumar B
f4ae00c530 HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena." 2019-02-26 00:36:14 +05:30
Yongjun Zhang
f7a27cdee4 HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li. 2019-02-23 09:35:36 -08:00
Sahil Takiar
f19c844e75 HDFS-14285. libhdfs hdfsRead copies entire array even if its only partially filled. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-22 17:54:32 -08:00
Erik Krogen
bad3ffd290 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-02-22 13:58:49 -08:00
Surendra Singh Lilhore
7d3b567194 HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi. 2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 2019-02-21 20:36:34 +05:30
Shweta Yakkali
1bea785020 HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 14:28:37 -08:00
Inigo Goiri
f5b4e0f971 HDFS-14302. Refactor NameNodeWebHdfsMethods#generateDelegationToken() to allow better extensibility. Contributed by CR Hota. 2019-02-20 13:55:13 -08:00
Sahil Takiar
a30059bb61 HDFS-14267. Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-20 11:38:37 -08:00
Surendra Singh Lilhore
41e18feda3 HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 2019-02-20 16:56:10 +05:30
Bharat Viswanadham
51950f149e Logging stale datanode information. Contributed by Karthik Palanisamy. 2019-02-19 17:01:07 -08:00
Kitti Nanasi
14282e311b HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-19 12:04:56 -08:00
Inigo Goiri
7587f97127 HDFS-14296. Prefer ArrayList over LinkedList in VolumeScanner. Contributed by BELUGA BEHR. 2019-02-18 17:55:43 -08:00
Inigo Goiri
1de25d134f HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR. 2019-02-18 15:30:38 -08:00
Inigo Goiri
235e3da90a HDFS-14287. DataXceiverServer May Double-Close PeerServer. Contributed by BELUGA BEHR. 2019-02-18 11:00:04 -08:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Inigo Goiri
dde0ab55aa HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR. 2019-02-15 16:32:27 -08:00
Anu Engineer
75e15cc0c4 HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects. 
Contributed by Elek, Marton.
 2019-02-14 23:33:25 -08:00
Erik Krogen
64f28f9efa HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5 HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena. 2019-02-14 22:13:14 +05:30
Shweta Yakkali
080a421911 HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-02-14 08:05:06 -08:00
Chen Liang
024c87291c HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-02-13 12:40:31 -08:00
Surendra Singh Lilhore
29b411d5f0 HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 2019-02-13 23:41:27 +05:30
Surendra Singh Lilhore
35d4f32b32 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 2019-02-13 23:06:46 +05:30
Vinayakumar B
7806403842 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:57:57 +05:30
Siyao Meng
5c10630ad8 HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-11 11:02:32 -08:00
Inigo Goiri
0ceb1b70f3 HDFS-14260. Replace synchronized method in BlockReceiver with atomic value. Contributed by BELUGA BEHR. 2019-02-11 10:09:44 -08:00
Yiqun Lin
177131793a HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li. 2019-02-08 20:51:30 +08:00
Surendra Singh Lilhore
4be87353e3 HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-07 16:51:59 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Siyao Meng
6aa63452b3 HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-06 12:19:21 -08:00
Erik Krogen
d3de8e162b HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-02-06 09:54:47 -08:00
Kitti Nanasi
49ddd8a6ed HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-05 15:14:19 -08:00
Wangda Tan
308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Kitti Nanasi
5f15a60e32 HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-02-04 14:52:07 -08:00
Inigo Goiri
0e79a86582 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 2019-02-04 11:59:48 -08:00
tiwalter
9aa3dc872c
HDFS-14158. Checkpointer ignores configured time period > 5 minutes 
This closes #449

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-02-04 13:51:07 +09:00
Takanobu Asanuma
ec77e95bbd HDFS-14232. libhdfs is not included in binary tarball. Contributed by Akira Ajisaka. 2019-02-04 10:12:09 +09:00
Kitti Nanasi
0ab7fc9200 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-01-31 16:07:24 -08:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Wei-Chiu Chuang
d1714c20e9 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1d523279da.
 2019-01-29 15:43:09 -08:00