big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,772 Commits 2 Branches 2 Tags 264 MiB
032ccba67c
Commit Graph

5812 Commits

Author SHA1 Message Date
Surendra Singh Lilhore
336abbd873 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 2019-10-14 23:31:43 +05:30
Ashvin
fabd41fa48 HDFS-14856. Fetch file ACLs while mounting external store. (#1478) 2019-10-14 09:44:56 -07:00
Ayush Saxena
91320b4461 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:14:14 +05:30
Eric Yang
aa24add8f0 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle
 2019-10-04 13:13:10 -04:00
Takanobu Asanuma
b23bdaf085 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 2019-10-04 16:47:06 +09:00
Stephen O'Donnell
c99a12167f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 22:13:50 -07:00
Virajith Jalaparti
844b766da5 HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. (#1573)" 2019-10-03 21:32:48 -07:00
Ayush Saxena
a3fe404244 HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C. 2019-10-03 22:34:55 +05:30
Chen Liang
1303255aee HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:38:20 -07:00
Akira Ajisaka
3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li. 2019-10-02 11:03:16 +09:00
Wei-Chiu Chuang
6ef6594c7e HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
 2019-10-01 21:16:46 +05:30
Konstantin V Shvachko
b3275ab1f2 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 16:48:10 -07:00
Konstantin V Shvachko
760b523e58 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-29 13:11:31 -07:00
Ayush Saxena
c4c8d5fd0e HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-28 21:44:44 +05:30
Ayush Saxena
0d5d0b914a Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao." 
This reverts commit ce58c05f1d.
 2019-09-28 21:32:15 +05:30
Ayush Saxena
ce58c05f1d HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 2019-09-27 18:58:19 +05:30
Ayush Saxena
66400c1cbb HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 2019-09-25 01:31:15 +05:30
Ayush Saxena
eb96a3093e HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:16:30 +05:30
Inigo Goiri
afa1006a53 HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor. 2019-09-24 09:58:42 -07:00
Ayush Saxena
c30e495557 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-23 21:22:50 +05:30
Vinayakumar B
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494). Contributed by Vinayakumar B. 2019-09-23 12:37:47 +05:30
Rakesh Radhakrishnan
659c88801d HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 2019-09-22 22:02:54 +05:30
Inigo Goiri
3f223bebfa HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun. 2019-09-20 11:45:01 -07:00
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Chen Liang
4ed0aefe9f HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 16:47:46 -07:00
Wei-Chiu Chuang
5dd859a8a0 HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-09-18 09:34:01 -07:00
Inigo Goiri
f580a87079 HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun. 2019-09-17 14:55:34 -07:00
Ayush Saxena
e04b8a46c3 HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina. 2019-09-14 10:39:20 +05:30
Stephen O'Donnell
06ad6540b3 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-13 10:36:45 -07:00
Surendra Singh Lilhore
4852a90e4b HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 2019-09-13 00:17:13 +05:30
Surendra Singh Lilhore
2ff2a7f612 HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina. 2019-09-12 19:43:18 +05:30
Surendra Singh Lilhore
d1c303a497 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 2019-09-12 19:11:50 +05:30
Akira Ajisaka
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor. 2019-09-12 12:41:57 +09:00
Surendra Singh Lilhore
d98c54816d HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green. 2019-09-06 18:12:42 +05:30
Surendra Singh Lilhore
fa7f03fc56 HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C. 2019-09-06 17:55:30 +05:30
Inigo Goiri
72d8b92ba5 HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun. 2019-09-05 11:44:02 -07:00
Xiaoyu Yao
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 2019-09-03 09:29:58 -07:00
Ayush Saxena
91b01a1db7 HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena. 2019-09-03 18:08:42 +05:30
Surendra Singh Lilhore
5ff76cb8bc HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 2019-09-03 12:37:09 +05:30
Stephen O'Donnell
915cbc91c0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-02 09:47:04 -07:00
Surendra Singh Lilhore
96f7dc1992 HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui. 2019-08-30 14:31:05 +05:30
Ayush Saxena
e220dac15c HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:40:13 +05:30
leosunli
915271245b Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-08-29 18:35:26 -07:00
Istvan Fajth
fcb7884bfc HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-29 16:56:14 -07:00
Inigo Goiri
3b22fcd377 HDFS-14256. Review Logging of NameNode Class. Contributed by David Mollitor. 2019-08-29 14:21:09 -07:00
Inigo Goiri
ffca734c62 HDFS-14104. Review getImageTxIdToRetain. Contributed by David Mollitor. 2019-08-29 14:18:35 -07:00
He Xiaoqiao
f600fbb6c4 HDFS-11246. FSNameSystem#logAuditEvent should be called outside the read or write locks. Contributed by He Xiaoqiao, Kuhu Shukla. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Kuhu Shukla <kshukla@apache.org>
 2019-08-29 10:10:52 -07:00
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
CR Hota
6e37d65b03 HDFS-14760. Log INFO mode if snapshot usage and actual usage differ. Contributed by CR Hota. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 16:20:25 -07:00
He Xiaoqiao
dde9399b37 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-27 15:26:59 -07:00
Shashikant Banerjee
ce8eb1283a Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee." 
This reverts commit 62445021d5 as it has unintended changes in DirectoryWithSnapshotFeature class..
 2019-08-27 23:23:44 +05:30
Arpit Agarwal
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 2019-08-26 15:57:55 -07:00
Chen Liang
ebef99dcf4 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:18:43 -07:00
Erik Krogen
d699022fce HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 2019-08-23 11:33:21 -07:00
Akira Ajisaka
bd7baea5a5
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 2019-08-23 15:28:27 +09:00
Akira Ajisaka
abc8fde4ca
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 2019-08-23 14:44:34 +09:00
Stephen O'Donnell
b67812ea21 HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index (#1028). Contributed by Stephen O'Donnell. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-08-22 17:09:57 -07:00
Stephen O'Donnell
93daf69f90 HDFS-14675. Increase Balancer Defaults Further. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Gabor Bota <gabota@apache.org>
 2019-08-22 15:21:33 -07:00
Surendra Singh Lilhore
76790a1e67 HDFS-14358. Provide LiveNode and DeadNode filter in DataNode UI. Contributed by hemanthboyina. 2019-08-22 12:27:45 +05:30
Ayush Saxena
0f598aed13 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 02:25:06 +05:30
Wei-Chiu Chuang
57f7370174 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit b58a35f374.
 2019-08-21 08:59:49 -07:00
Surendra Singh Lilhore
3a145e2918 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-20 15:55:26 -07:00
Sandeep Kumar
269b543367 HADOOP-16523. Minor spell mistake in comment : Line number 101 (#388) 2019-08-20 13:36:23 -07:00
Wei-Chiu Chuang
4cb22cd867 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:37:11 -07:00
Sunil G
bd92462321 HDFS-14729. Upgrade Bootstrap and jQuery versions used in HDFS UIs. Contributed by Vivek Ratnavel Subramanian. This closes #1297 2019-08-20 18:42:19 +05:30
Shashikant Banerjee
62445021d5 HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee. 2019-08-20 14:37:01 +05:30
Wei-Chiu Chuang
360a96f342 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 2019-08-19 13:08:55 -07:00
Surendra Singh Lilhore
b8db5b9a98 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 18:13:35 -07:00
Wei-Chiu Chuang
b58a35f374 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:46:00 -07:00
Wei-Chiu Chuang
d873ddd656 HDFS-14127. Add a description about the observer read configuration. Contributed by xiangheng. 2019-08-17 04:39:22 -07:00
Wei-Chiu Chuang
528378784f HDFS-12012. Fix spelling mistakes in BPServiceActor.java. Contributed by chencan. 2019-08-17 04:37:05 -07:00
hunshenshi
a38b9e137e HDFS-14456:HAState#prepareToEnterState neednt a lock (#770) Contributed by hunshenshi. 2019-08-16 14:53:06 -07:00
LeonGao91
9a1d8cfaf5 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 2019-08-16 08:00:51 -07:00
Surendra Singh Lilhore
34681643e9 HDFS-14719. Correct the safemode threshold value in BlockManagerSafeMode. Contributed by hemanthboyina. 2019-08-15 12:12:18 +05:30
Shashikant Banerjee
0a85af959c HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 2019-08-15 10:16:25 +05:30
Wei-Chiu Chuang
c13ec7ab66 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 2019-08-13 17:15:26 -07:00
Wei-Chiu Chuang
633b7c1cfe HDFS-14625. Make DefaultAuditLogger class in FSnamesystem to Abstract. Contributed by hemanthboyina. 2019-08-13 16:50:49 -07:00
Masatake Iwasaki
da0006fe04 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-14 08:39:40 +09:00
Ayush Saxena
e9b6b81de4 HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng. 2019-08-13 19:17:10 +05:30
Siyao Meng
c92b49876a HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 2019-08-12 17:26:20 -07:00
Yiqun Lin
8a77a224c7 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-09 18:40:28 -07:00
Wei-Chiu Chuang
ce3c5a3e3b HDFS-14623. In NameNode Web UI, for Head the file (first 32K) old data is showing. Contributed by hemanthboyina. 2019-08-09 15:41:37 -07:00
Wei-Chiu Chuang
865021b8c9 HDFS-14195. OIV: print out storage policy id in oiv Delimited output. Contributed by Wang, Xinglong. 2019-08-09 15:37:29 -07:00
Rakesh Radhakrishnan
f6fa865d6f HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 2019-08-09 14:07:54 +05:30
Wei-Chiu Chuang
6ad9a11494 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 2019-08-08 13:50:30 -07:00
Stephen O'Donnell
b0799148cf HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-08 13:36:39 -07:00
Ayush Saxena
11f750e6a7 HDFS-14616. Add the warn log when the volume available space isn't enough. Contributed by liying. 2019-08-08 00:49:58 +05:30
Ayush Saxena
de64253892 HDFS-14608. DataNode#DataTransfer should be named. Contributed by Inigo Goiri. 2019-08-08 00:37:56 +05:30
Erik Krogen
827dbb11e2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 2019-08-07 09:25:58 -07:00
Yiqun Lin
a5bb1e8ee8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 10:18:11 +08:00
Lokesh Jain
954ff36360 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 15:30:50 -07:00
Stephen O'Donnell
35e0a01d7b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 07:57:46 -07:00
Ayush Saxena
11272159bb HDFS-14679. Failed to add erasure code policies with example template. Contributed by Yuan Zhou. 2019-08-06 08:42:15 +05:30
Wei-Chiu Chuang
b964b81f85 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 2019-08-02 10:54:32 -07:00
Siyao Meng
99bf1dc9eb HDFS-14683. WebHDFS: Add erasureCodingPolicy field to GETCONTENTSUMMARY response (#1189) Contributed by Siyao Meng. 2019-08-01 17:14:07 -07:00
Wei-Chiu Chuang
32607dbd98 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-01 15:48:36 -07:00
Wei-Chiu Chuang
99f88c30cb HDFS-14419. Avoid repeated calls to the listOpenFiles function. Contributed by HuangTao. 2019-07-30 16:50:06 -07:00
Chao Sun
3ae775d740 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-07-30 16:01:17 -07:00
Erik Krogen
1f26cc8705 HDFS-13783. Add an option to the Balancer to make it run as a long-running service. Contributed by Chen Zhang. 2019-07-30 15:42:55 -07:00
Inigo Goiri
7849bdcf70 HDFS-14449. Expose total number of DT in JMX for Namenode. Contributed by Fengnan Li. 2019-07-30 13:45:27 -07:00
Wei-Chiu Chuang
8053085388 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:31:34 -07:00
Ayush Saxena
02bd02b5af HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-07-28 08:41:42 +05:30
Ayush Saxena
62deab17a3 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 2019-07-25 06:51:07 +05:30
Ayush Saxena
377f95bbe8 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 2019-07-23 08:37:55 +05:30
Chen Liang
5e6cc6fe8a HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-07-17 14:37:16 -07:00
Ayush Saxena
af0665c7f5 HDFS-13647. Fix the description of storageType option for space quota. Contributed by Takanobu Asanuma. 2019-07-17 23:22:32 +05:30
Ayush Saxena
f77d54c243 HDFS-14642. processMisReplicatedBlocks does not return correct processed count. Contributed by Stephen O'Donnell. 2019-07-16 08:14:27 +05:30
Rakesh Radhakrishnan
e98adb00b7 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 2019-07-15 13:02:37 +05:30
Shashikant Banerjee
f9fab9f22a HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 2019-07-12 15:42:06 +05:30
Chen Liang
8fb5ca3f40 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-07-11 13:23:25 -07:00
Inigo Goiri
5747f6cff5 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 2019-07-11 08:55:44 -07:00
Inigo Goiri
eccc9a40de HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-10 11:11:52 -07:00
Yiqun Lin
993dc8726b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 2019-07-10 10:53:34 +08:00
Erik Krogen
4632708148 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-08 14:46:25 -07:00
Wei-Chiu Chuang
ec851e4db2 HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 2019-07-08 15:23:46 +08:00
leosunli
9c90729486 HDFS-13694. Making md5 computing being in parallel with image loading. 2019-07-05 10:17:37 -07:00
Masatake Iwasaki
1c254a8ec7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 2019-07-04 23:20:42 +09:00
Weiwei Yang
729cb3aefe HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-04 09:37:24 +08:00
paulward24
d8bac50e12 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 2019-07-01 13:54:49 -07:00
paulward24
d203045c30 HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 2019-06-28 16:06:43 -07:00
Inigo Goiri
4a212242d9 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 2019-06-27 10:00:05 -07:00
Anu Engineer
041e7a7dee HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.
 2019-06-24 19:07:22 -07:00
Giovanni Matteo Fumarola
2a2d5eb441 HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Wei-Chiu Chuang
645d67bc4f HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 2019-06-21 18:27:03 -07:00
Wei-Chiu Chuang
1524e2e6c5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 2019-06-21 18:17:36 -07:00
Lokesh Jain
272b96d243 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:23:05 -07:00
Surendra Singh Lilhore
5962a518bd HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 09:40:29 -07:00
Wei-Chiu Chuang
71ecd2e411 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-19 10:27:53 -07:00
David Mollitor
f5ecc0bc08 HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-19 10:06:40 -07:00
Wei-Chiu Chuang
48e564f7e2 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-18 22:28:21 -07:00
Bharat Viswanadham
f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya. 2019-06-18 21:58:37 -07:00
Elek, Marton
81ec909411 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 12:01:26 -07:00
Inigo Goiri
3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Wei-Chiu Chuang
335c1c9938 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 2019-06-18 00:35:48 -07:00
Weiwei Yang
e3172221ea HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 21:48:16 -07:00
Santosh Marella
6822193ee6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:20:30 -07:00
Wei-Chiu Chuang
7314185c4a Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit ae4143a529.
 2019-06-17 16:18:48 -07:00
Wei-Chiu Chuang
f9a7b442fd HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-17 15:18:17 -07:00
Adam Antal
8370a0ae16 HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-15 06:47:10 -07:00
Giovanni Matteo Fumarola
b24efa11ea HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri. 2019-06-14 13:37:23 -07:00
Santosh Marella
ae4143a529 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 10:42:54 -07:00
dineshchitlangia
9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method 
Closes #967
 2019-06-14 16:09:37 +02:00
Inigo Goiri
4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Stephen O'Donnell
50de0874d0 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-12 10:30:23 -07:00
Eric Yang
4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Inigo Goiri
5740eea081 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-06-11 11:48:35 -07:00
Anu Engineer
101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
He Xiaoqiao
bd46bdf9f9 HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 17:21:52 -07:00
Hanisha Koneru
7217494f40 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 13:45:37 -07:00
Shweta Yakkali
d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Inigo Goiri
e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri
944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang
294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen
d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
Kihwal Lee
9122b9b649 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 2019-05-30 17:28:45 -05:00
He Xiaoqiao
33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali
6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena
c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Eric Yang
d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Rakesh Radhakrishnan
37900c5639 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 2019-05-26 14:30:11 +05:30
Erik Krogen
55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Inigo Goiri
c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian
f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Inigo Goiri
1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Virajith Jalaparti
77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Rakesh Radhakrishnan
9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore
69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Surendra Singh Lilhore
e424392a62 HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi. 2019-05-05 13:38:09 +05:30
Giovanni Matteo Fumarola
7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Inigo Goiri
865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Siyao Meng
7cb46f035a HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-01 16:48:44 -07:00
Arpit Agarwal
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 2019-04-29 14:49:35 -07:00
Puleya7
13907d8479 HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-04-17 21:39:06 +09:00
Inigo Goiri
be6c8014e6 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 2019-04-16 10:34:31 -07:00
Chen Liang
626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Adam Antal
586826fe99
HDFS-12245. Fix INodeId javadoc 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-11 10:01:47 +09:00
Shweta
e9b859f749 HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-05 16:09:00 -07:00
Akira Ajisaka
67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 2019-04-05 12:29:11 +09:00
Stephen O'Donnell
d2637cb176 HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-04 04:07:18 -07:00
Inigo Goiri
7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Gabor Bota
d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Wei-Chiu Chuang
be488b6070 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-03 11:00:59 -07:00
Uma Maheswara Rao G
35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan
f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Inigo Goiri
15d38b1bf9 HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor. 2019-03-28 03:37:33 -07:00
Takanobu Asanuma
55fb3c32fb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 2019-03-27 03:27:02 +09:00
Stephen O'Donnell
3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Inigo Goiri
ff06ef0631 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 2019-03-15 11:06:42 -07:00
Uma Maheswara Rao G
ba50a36a3e HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 2019-03-14 22:21:08 -07:00
Erik Krogen
66357574ae HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 2019-03-13 13:15:56 -07:00
Stephen O'Donnell
34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen
1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Wei-Chiu Chuang
6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri
945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B
f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Inigo Goiri
4b7313e640 HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker. 2019-03-04 23:03:32 -08:00
Wei-Chiu Chuang
387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00
Yiqun Lin
18ea0c1493 HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang. 2019-03-04 17:34:24 +08:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00