big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,668 Commits 2 Branches 2 Tags 264 MiB
93daf69f90
Commit Graph

5605 Commits

Author SHA1 Message Date
Rakesh Radhakrishnan
e98adb00b7 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 2019-07-15 13:02:37 +05:30
Shashikant Banerjee
f9fab9f22a HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 2019-07-12 15:42:06 +05:30
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Inigo Goiri
5747f6cff5 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 2019-07-11 08:55:44 -07:00
Inigo Goiri
eccc9a40de HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-10 11:11:52 -07:00
Yiqun Lin
993dc8726b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 2019-07-10 10:53:34 +08:00
Erik Krogen
4632708148 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-08 14:46:25 -07:00
Wei-Chiu Chuang
ec851e4db2 HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 2019-07-08 15:23:46 +08:00
leosunli
9c90729486 HDFS-13694. Making md5 computing being in parallel with image loading. 2019-07-05 10:17:37 -07:00
Masatake Iwasaki
1c254a8ec7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 2019-07-04 23:20:42 +09:00
Weiwei Yang
729cb3aefe HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-04 09:37:24 +08:00
paulward24
d8bac50e12 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 2019-07-01 13:54:49 -07:00
paulward24
d203045c30 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 2019-06-28 16:06:43 -07:00
Inigo Goiri
4a212242d9 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 2019-06-27 10:00:05 -07:00
Anu Engineer
041e7a7dee HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.
 2019-06-24 19:07:22 -07:00
Giovanni Matteo Fumarola
2a2d5eb441 HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Wei-Chiu Chuang
645d67bc4f HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 2019-06-21 18:27:03 -07:00
Wei-Chiu Chuang
1524e2e6c5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 2019-06-21 18:17:36 -07:00
Lokesh Jain
272b96d243 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:23:05 -07:00
Surendra Singh Lilhore
5962a518bd HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 09:40:29 -07:00
Wei-Chiu Chuang
71ecd2e411 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-19 10:27:53 -07:00
David Mollitor
f5ecc0bc08 HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-19 10:06:40 -07:00
Wei-Chiu Chuang
48e564f7e2 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-18 22:28:21 -07:00
Bharat Viswanadham
f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya. 2019-06-18 21:58:37 -07:00
Elek, Marton
81ec909411 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 12:01:26 -07:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Wei-Chiu Chuang
335c1c9938 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 2019-06-18 00:35:48 -07:00
Weiwei Yang
e3172221ea HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 21:48:16 -07:00
Santosh Marella
6822193ee6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:20:30 -07:00
Wei-Chiu Chuang
7314185c4a Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit ae4143a529.
 2019-06-17 16:18:48 -07:00
Wei-Chiu Chuang
f9a7b442fd HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-17 15:18:17 -07:00
Adam Antal
8370a0ae16 HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-15 06:47:10 -07:00
Giovanni Matteo Fumarola
b24efa11ea HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri. 2019-06-14 13:37:23 -07:00
Santosh Marella
ae4143a529 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 10:42:54 -07:00
dineshchitlangia
9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method 
Closes #967
 2019-06-14 16:09:37 +02:00
Inigo Goiri
4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Stephen O'Donnell
50de0874d0 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-12 10:30:23 -07:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Inigo Goiri
5740eea081 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-06-11 11:48:35 -07:00
Anu Engineer
101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
He Xiaoqiao
bd46bdf9f9 HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 17:21:52 -07:00
Hanisha Koneru
7217494f40 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 13:45:37 -07:00
Shweta Yakkali
d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Inigo Goiri
e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri
944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen
d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Kihwal Lee
9122b9b649 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 2019-05-30 17:28:45 -05:00
He Xiaoqiao
33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena
c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Eric Yang
d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Rakesh Radhakrishnan
37900c5639 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 2019-05-26 14:30:11 +05:30
Erik Krogen
55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Inigo Goiri
c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Inigo Goiri
1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Virajith Jalaparti
77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Rakesh Radhakrishnan
9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore
69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Surendra Singh Lilhore
e424392a62 HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi. 2019-05-05 13:38:09 +05:30
Giovanni Matteo Fumarola
7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Inigo Goiri
865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Siyao Meng
7cb46f035a HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-01 16:48:44 -07:00
Arpit Agarwal
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 2019-04-29 14:49:35 -07:00
Puleya7
13907d8479 HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-04-17 21:39:06 +09:00
Inigo Goiri
be6c8014e6 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 2019-04-16 10:34:31 -07:00
Chen Liang
626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Adam Antal
586826fe99
HDFS-12245. Fix INodeId javadoc 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-11 10:01:47 +09:00
Shweta
e9b859f749 HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-05 16:09:00 -07:00
Akira Ajisaka
67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 2019-04-05 12:29:11 +09:00
Stephen O'Donnell
d2637cb176 HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-04 04:07:18 -07:00
Inigo Goiri
7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Wei-Chiu Chuang
be488b6070 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-03 11:00:59 -07:00
Uma Maheswara Rao G
35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan
f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Inigo Goiri
15d38b1bf9 HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor. 2019-03-28 03:37:33 -07:00
Takanobu Asanuma
55fb3c32fb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 2019-03-27 03:27:02 +09:00
Stephen O'Donnell
3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Inigo Goiri
ff06ef0631 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 2019-03-15 11:06:42 -07:00
Uma Maheswara Rao G
ba50a36a3e HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 2019-03-14 22:21:08 -07:00
Erik Krogen
66357574ae HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 2019-03-13 13:15:56 -07:00
Stephen O'Donnell
34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen
1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Wei-Chiu Chuang
6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri
945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B
f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Inigo Goiri
4b7313e640 HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker. 2019-03-04 23:03:32 -08:00
Wei-Chiu Chuang
387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00
Yiqun Lin
18ea0c1493 HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang. 2019-03-04 17:34:24 +08:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri
80b77deb42 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 2019-03-01 09:18:51 -08:00
Brahma Reddy Battula
7a0db2f92b HDFS-13997. Secondary NN Web UI displays nothing, and the console log shows moment is not defined. Contributed by Ayush Saxena 2019-03-01 12:24:52 +05:30
Erik Krogen
0feba4396f HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-02-28 08:48:10 -08:00
Vinayakumar B
f4ae00c530 HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena." 2019-02-26 00:36:14 +05:30
Yongjun Zhang
f7a27cdee4 HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li. 2019-02-23 09:35:36 -08:00
Surendra Singh Lilhore
7d3b567194 HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi. 2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 2019-02-21 20:36:34 +05:30