Commit Graph

7891 Commits

Author SHA1 Message Date
jianghuazhu
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823)
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 17:24:12 +09:00
KevinWikant
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675)
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 22:59:08 +09:00
secfree
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
Jackson Wang
9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. (#3803)
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
2021-12-21 11:04:02 +09:00
jianghuazhu
746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 2021-12-20 11:28:55 +00:00
secfree
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
liubingxing
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-17 13:31:28 +08:00
litao
22f5e1885d
HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-16 13:49:50 +09:00
litao
10cdde59b4
HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 2021-12-16 13:29:32 +09:00
litao
59c650802d
HDFS-16378. Add datanode address to BlockReportLeaseManager logs (#3786). Contributed by tomscut.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-15 11:16:32 +08:00
Ayush Saxena
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-14 17:21:51 +09:00
litao
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 2021-12-14 11:19:06 +09:00
qinyuren
35556ea043
HDFS-16333. fix balancer bug when transfer an EC block (#3679) 2021-12-09 13:24:09 +09:00
PHILO-HE
1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761) 2021-12-08 17:52:40 +05:30
GuoPhilipse
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode (#3661)
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 21:21:48 +09:00
GuoPhilipse
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse.
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-08 16:34:54 +05:30
Masatake Iwasaki
e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740) 2021-12-07 21:39:04 +09:00
GuoPhilipse
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem (#3713). Contributed by guophilipse.
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-07 14:19:27 +05:30
GuoPhilipse
1509ea5de2
HDFS-16338. Correct fsimage error configuration message (#3684). Contributed by guophilipse.
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-06 14:00:50 +05:30
Takanobu Asanuma
e8e69de106 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 2021-12-06 10:44:36 +09:00
litao
c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo (#3747). Contributed by tomscut.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-04 22:54:44 +05:30
huhaiyang
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664)
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-04 02:19:36 +09:00
bitterfox
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake (#3677)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-03 23:30:13 +09:00
wangzhaohui
0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 (#3742) 2021-12-03 16:51:04 +05:30
litao
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-03 14:12:05 +09:00
litao
1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand (#3732). Contributed by tomscut. 2021-12-01 12:48:21 +05:30
litao
829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-29 17:48:10 +09:00
litao
86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-26 19:52:14 +09:00
Takanobu Asanuma
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-25 14:05:04 +09:00
Viraj Jasani
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 2021-11-25 10:23:04 +08:00
huhaiyang
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Viraj Jasani
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 22:54:11 +09:00
Mukul Kumar Singh
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 2021-11-23 13:36:14 +00:00
litao
0ed817babb
HDFS-16337. Show start time of Datanode on Web (#3682)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2021-11-23 10:18:30 +08:00
Symious
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
litao
20b78c8f97
HDFS-16335. Fix HDFSCommands.md (#3681) 2021-11-22 17:07:52 +09:00
Hui Fei
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
GuoPhilipse
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description (#3678)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-22 07:41:57 +09:00
litao
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer (#3663)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-11-19 11:37:10 +09:00
GuoPhilipse
0c1eb0125b
HDFS-16328. Correct disk balancer desc (#3669)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 01:02:01 +09:00
Viraj Jasani
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:55:39 +09:00
litao
dc751df63b
HDFS-16329. Fix log format for BlockManager (#3670)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:42:01 +09:00
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
huhaiyang
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Viraj Jasani
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 2021-11-17 22:17:09 +09:00
Neil
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643)
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
GuoPhilipse
6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-14 19:59:15 +05:30
huhaiyang
db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00