big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,068 Commits 2 Branches 2 Tags
Commit Graph

9267 Commits

Author SHA1 Message Date
litao
b5d2e00f81 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a351fa1a5dd668c318b20437e6aaa35b)
 2022-01-27 13:44:03 +09:00
litao
ef1a2b478b HDFS-16398. Reconfig block report parameters for datanode (#3831) 
(cherry picked from commit c2ff39006f5deca60a2a2524707350e06bba2779)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2022-01-26 17:36:17 +09:00
Wei-Chiu Chuang
ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages (#3883) (#3924) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e63a4377331643eddedcab79d4a51468)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java

Co-authored-by: qinyuren <1476659627@qq.com>
 2022-01-26 11:54:13 +08:00
Bryan Beaudreault
bd13d73334 HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 
(cherry picked from commit 94b884ae553e9d67f57e3ae7d2b22468c6f342fa)
 2022-01-25 11:43:47 +00:00
daimin
728ed10a7c
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1ed49e06cc8973412952e09ed08bb9c0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
 2022-01-25 09:58:20 +08:00
liubingxing
d6ff60df65
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52ada03e099fa7dbb3efe62345ca27588)
 2022-01-20 18:47:00 +08:00
litao
f9c0bc094a HDFS-16399. Reconfig cache report parameters for datanode (#3841) 
(cherry picked from commit e3556463301f103ccf26a17e116c507c143e7121)
 2022-01-19 18:43:15 +09:00
litao
11fe5279b0 HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit f02374df9236d6498a549cc6bfa07bffff5ef7d8)
 2022-01-19 18:42:48 +09:00
litao
cdaf4d89f9 HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 52ec65fd1022a72dfdfefaf50489dc61bb3fc8f7)
 2022-01-19 18:40:41 +09:00
Viraj Jasani
831c11c47a HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit b038042ece550c34170d4958f84e91a1708761a5)
 2022-01-19 16:01:00 +09:00
qinyuren
1c71d6e9fe HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699bd1360c109c18df0261897afea05ba)
 2022-01-19 13:44:02 +09:00
Xiangyi Zhu
b5e7f59e53
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 23:18:05 +08:00
Jackson Wang
926222a0d0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d22dbba8facb94e80333cea5ffeed5ba)
 2022-01-14 22:40:58 +09:00
Ashutosh Gupta
6b83fe4a00 HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:00:23 -08:00
liever18
3a82899493 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5851d3cdc3b125daebe5cc9305ad6f0)
 2022-01-05 15:46:21 +00:00
Wei-Chiu Chuang
350b51f287 Make upstream aware of 3.3.1 release 2022-01-04 14:48:49 -08:00
Ashutosh Gupta
bad3a0964c HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2d79c3691bee29cb3fc4123062e5f7a)
 2022-01-04 13:25:56 +09:00
jianghuazhu
b623acff5a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2021-12-24 17:25:00 +09:00
wangzhaohui
cc7b7e174c HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629c2db33f03c80e93c7e54fceedbfbe6)
 2021-12-21 11:46:23 +00:00
jianghuazhu
a4496a0187 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2021-12-20 11:32:01 +00:00
litao
e60b8a0358 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2021-12-16 13:51:40 +09:00
litao
75af1d67e4 HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4666e7f70e0788409ca7b214539cedb)
 2021-12-16 13:31:17 +09:00
Ayush Saxena
37ced452d9 HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:23:24 +09:00
litao
8559ba46c2 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b561c9aacb5e74bd20195a817d5fd8d)
 2021-12-14 11:53:46 +09:00
GuoPhilipse
2720479988 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7430692d1f40f59e9cb24690cbda216)
 2021-12-10 16:33:33 +09:00
Akira Ajisaka
35c5c6bb83 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2021-12-10 01:38:26 +09:00
qinyuren
0dfb4eb602
HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:33:03 +09:00
Takanobu Asanuma
a67f4dc71a Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd533f0319e31357f037939339a6c156.
 2021-12-09 16:36:54 +09:00
qinyuren
55c0e676fd HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043c063f548b81ad431d9452f010b18f7)
 2021-12-09 13:26:00 +09:00
lzx404243
1ee661d7da HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c123544e1c1cbee759a11879af3b19260b3)
 2021-12-08 20:46:22 +09:00
LeonGao
63495978b2 HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661d694fb0775f4f84333c1317f4e6899)
 2021-12-06 16:18:25 +09:00
Takanobu Asanuma
f6b3361b4b HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106c03d041a0d280ea727e3f252460163)
 2021-12-06 11:05:54 +09:00
huhaiyang
a24c98163f
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b402b4bdf5b7771f5124790fc866965)
 2021-12-04 02:27:35 +09:00
huhaiyang
bcbe4fbfa7
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411ebee11372314e82d7ea0606c348d014)
 2021-12-04 02:24:40 +09:00
bitterfox
2cdecd420a
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:35:46 +09:00
litao
f5e15dfb11 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4240347675e20df0ce047245f465c7b)
 2021-11-29 17:54:58 +09:00
litao
b428d534fb HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82208b6b04551e637414d0d9d64a4173)
 2021-11-26 19:53:34 +09:00
Akira Ajisaka
4a032bc88d Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794657a77550318c83d3d7303d745740b.
 2021-11-25 14:24:30 +09:00
Viraj Jasani
6094e1ec9a
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523941622a140fd877f06e9b59f48c48b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2021-11-25 14:15:10 +09:00
Viraj Jasani
a9125bba24 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b619159dc4bf185ad53b950bdf5de52a)
 2021-11-25 10:28:40 +08:00
Ayush Saxena
3b79e0f794
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2021-11-25 01:54:14 +05:30
Viraj Jasani
450efaeabe
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b38129bcef687506742b3d7d35e8e4947a9)
 2021-11-24 22:56:23 +09:00
Mukul Kumar Singh
97de402970 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1f708b7c3a0701886fa63068327e9b3)
 2021-11-23 13:39:16 +00:00
litao
d746e2cdd8
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babbcaa0560c11e2065cff3fff0bf328c5)
 2021-11-23 15:43:02 +08:00
litao
90085eb4cb HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f978dc74f03a0dd5551904f8ec40ccf82)
 2021-11-22 17:14:17 +09:00
GuoPhilipse
6b78a753e3
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc1799e95feced466411743e9f9fddd4)
 2021-11-22 07:45:42 +09:00
litao
166c38bcbf HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e90f7ca7cf6d1418893340338eb8760f)
 2021-11-19 11:39:06 +09:00
GuoPhilipse
8166bcc668
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b1e0c8bdc3f4170295771d75c5ed776)
 2021-11-19 01:02:31 +09:00
Viraj Jasani
2f587bf15a
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e118711852c58e9f0ebc3493fcb5ebeb)
 2021-11-19 00:56:13 +09:00
litao
4036213273
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b4ab2c9c26a1efe7479c31fd1de80d5)
 2021-11-19 00:43:25 +09:00