big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,801 Commits 2 Branches 2 Tags 264 MiB
4c57fb4d6b
Commit Graph

9174 Commits

Author SHA1 Message Date
Takanobu Asanuma
4c57fb4d6b
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. (#3992) 
(cherry picked from commit 1cd96e8dd8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-16 09:42:43 +09:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
singer-bin
ce7cabb771
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
 2022-02-06 13:36:57 +08:00
daimin
709e617a84
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
 2022-02-06 13:06:35 +08:00
KevinWikant
7171e2190e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)
 2022-01-31 13:19:40 +09:00
KevinWikant
5e2eac6c41
HDFS-16303. Improve handling of datanode lost while decommissioning (#3921) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:18:36 +09:00
secfree
110104da38 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:05:32 +09:00
litao
b5d2e00f81 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:44:03 +09:00
litao
ef1a2b478b HDFS-16398. Reconfig block report parameters for datanode (#3831) 
(cherry picked from commit c2ff39006f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2022-01-26 17:36:17 +09:00
Wei-Chiu Chuang
ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages (#3883) (#3924) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java

Co-authored-by: qinyuren <1476659627@qq.com>
 2022-01-26 11:54:13 +08:00
Bryan Beaudreault
bd13d73334 HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 
(cherry picked from commit 94b884ae55)
 2022-01-25 11:43:47 +00:00
daimin
728ed10a7c
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
 2022-01-25 09:58:20 +08:00
liubingxing
d6ff60df65
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
 2022-01-20 18:47:00 +08:00
litao
f9c0bc094a HDFS-16399. Reconfig cache report parameters for datanode (#3841) 
(cherry picked from commit e355646330)
 2022-01-19 18:43:15 +09:00
litao
11fe5279b0 HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit f02374df92)
 2022-01-19 18:42:48 +09:00
litao
cdaf4d89f9 HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 52ec65fd10)
 2022-01-19 18:40:41 +09:00
Viraj Jasani
831c11c47a HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit b038042ece)
 2022-01-19 16:01:00 +09:00
qinyuren
1c71d6e9fe HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:02 +09:00
Xiangyi Zhu
b5e7f59e53
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 23:18:05 +08:00
Jackson Wang
926222a0d0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:40:58 +09:00
Ashutosh Gupta
6b83fe4a00 HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:00:23 -08:00
liever18
3a82899493 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 15:46:21 +00:00
Wei-Chiu Chuang
350b51f287 Make upstream aware of 3.3.1 release 2022-01-04 14:48:49 -08:00
Ashutosh Gupta
bad3a0964c HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:25:56 +09:00
jianghuazhu
b623acff5a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:25:00 +09:00
wangzhaohui
cc7b7e174c HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 11:46:23 +00:00
jianghuazhu
a4496a0187 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 11:32:01 +00:00
litao
e60b8a0358 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:51:40 +09:00
litao
75af1d67e4 HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2021-12-16 13:31:17 +09:00
Ayush Saxena
37ced452d9 HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:23:24 +09:00
litao
8559ba46c2 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2021-12-14 11:53:46 +09:00
GuoPhilipse
2720479988 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2021-12-10 16:33:33 +09:00
Akira Ajisaka
35c5c6bb83 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2021-12-10 01:38:26 +09:00
qinyuren
0dfb4eb602
HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:33:03 +09:00
Takanobu Asanuma
a67f4dc71a Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2021-12-09 16:36:54 +09:00
qinyuren
55c0e676fd HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2021-12-09 13:26:00 +09:00
lzx404243
1ee661d7da HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c1235)
 2021-12-08 20:46:22 +09:00
LeonGao
63495978b2 HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2021-12-06 16:18:25 +09:00
Takanobu Asanuma
f6b3361b4b HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2021-12-06 11:05:54 +09:00
huhaiyang
a24c98163f
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2021-12-04 02:27:35 +09:00
huhaiyang
bcbe4fbfa7
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2021-12-04 02:24:40 +09:00
bitterfox
2cdecd420a
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:35:46 +09:00
litao
f5e15dfb11 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2021-11-29 17:54:58 +09:00
litao
b428d534fb HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2021-11-26 19:53:34 +09:00
Akira Ajisaka
4a032bc88d Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2021-11-25 14:24:30 +09:00
Viraj Jasani
6094e1ec9a
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2021-11-25 14:15:10 +09:00
Viraj Jasani
a9125bba24 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:28:40 +08:00
Ayush Saxena
3b79e0f794
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2021-11-25 01:54:14 +05:30
Viraj Jasani
450efaeabe
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:23 +09:00
Mukul Kumar Singh
97de402970 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:39:16 +00:00