hfutatzhanghb
7012986fc3
HDFS-17345. Add a metrics to record block report generating cost time. ( #6475 ). Contributed by farmmamba.
...
Reviewed-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-03-06 16:59:00 +08:00
dannytbecker
5584efd8d4
HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade ( #6583 )
2024-03-05 11:31:31 -08:00
Jtdellaringa
a2d7241190
HDFS-17404. Add NN Socket Address to log when processing command from active NN ( #6598 )
...
* Add nnSocketAddress to log when processing command from active nn
2024-03-05 10:33:22 -08:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. ( #6430 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-03-02 21:35:24 +09:00
huhaiyang
34bb28649c
HDFS-17406. Suppress UnresolvedPathException in hdfs router log ( #6603 )
2024-03-01 19:03:41 +08:00
hfutatzhanghb
15af52954f
HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. ( #6509 ). Contributed by farmmamba.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-02-27 10:19:57 +08:00
ZanderXu
a897e745f5
HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem ( #6567 )
2024-02-23 15:37:27 +08:00
Lars Francke
d1daf26b85
Fixes HDFS-17181 by routing all CREATE requests to the BlockManager ( #6108 )
2024-02-20 10:58:49 -08:00
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes ( #6508 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. ( #6504 ) Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-18 09:15:40 +08:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up ( #6498 )
2024-02-13 16:24:23 -08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. ( #4052 )
...
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607 ).
Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
2024-02-13 16:33:51 +00:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider ( #6510 )
...
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
2024-02-13 10:49:39 +09:00
huhaiyang
5ad7737132
HDFS-17342. Fix DataNode may invalidates normal block causing missing block ( #6464 ). Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Chengwei Wang <1139557635@qq.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-02-06 17:52:52 +08:00
Takanobu Asanuma
9a7eeadaac
HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf ( #6522 )
2024-02-06 11:54:14 +09:00
Alexander
20d8596af2
HDFS-17353. Fix failing RBF module tests. ( #6491 ) Contributed by Alexander Bogdanov
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:26:30 +08:00
huhaiyang
6464507cd1
HDFS-17369. Add uuid into datanode info for NameNodeMXBean ( #6521 ) Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:20:04 +08:00
hfutatzhanghb
4f4b846986
HDFS-17359. EC: recheck failed streamers should only after flushing all packets. ( #6503 ). Contributed by farmmamba.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-01 23:53:37 +09:00
Lei313
ac471d7daa
HDFS-17339:Skip cacheReport when one blockPool does not have CacheBlock on this DataNode ( #6456 ). Contributed by lei w.
...
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 21:15:20 +08:00
huhaiyang
caba9bbab3
HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt ( #6476 ). Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 12:49:18 +08:00
LiuGuH
8193a84020
HDFS-17324. RBF: Router should not return nameservices that not enable observer r… ( #6412 )
2024-01-23 12:30:08 -08:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. ( #6368 ). Contributed by farmmamba.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-22 11:50:51 +08:00
LiuGuH
2a1ee8dfcd
HDFS-17311. RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue. ( #6392 ) Contributed by liuguanghua.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:55:23 +08:00
slfan1989
15e1789baf
Revert "HDFS-16016. BPServiceActor to provide new thread to handle IBR ( #2998 )" ( #6457 ) Contributed by Shilun Fan.
...
This reverts commit c1bf3cb0
2024-01-20 07:51:55 +08:00
Jian Zhang
1036544480
HDFS-17302. RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation. ( #6380 )
2024-01-19 14:02:21 -08:00
slfan1989
8444f69511
Preparing for 3.5.0 development ( #6411 )
...
Co-authored-by: slfan1989 <slfan1989@apache.org>
2024-01-19 15:05:22 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException ( #6446 )
...
Print a warn log message for read retries and only print the full stack trace for a read request failure.
Contributed by: Xing Lin
2024-01-18 18:03:28 -08:00
Lei313
cc4c4be1b7
HDFS-17331:Fix Blocks are always -1 and DataNode version are always UNKNOWN in federationhealth.html ( #6429 ). Contributed by lei w.
...
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-18 21:10:54 +08:00
hfutatzhanghb
a30681077b
HDFS-17291. DataNode metric bytesWritten is not totally accurate in some situations. ( #6360 ). Contributed by farmmamba.
...
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-13 20:45:00 +08:00
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. ( #6357 ). Contributed by farmmamba.
...
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-13 20:34:02 +08:00
hfutatzhanghb
6a053765ee
HDFS-17312. packetsReceived metric should ignore heartbeat packet. ( #6394 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-01-11 22:08:37 +09:00
xuzifu666
99a59ae9e6
HDFS-17317. Improve the resource release for metaOut in DebugAdmin ( #6402 ). Contributed by xy.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
2024-01-07 00:59:31 +05:30
huangzhaobo
08713665c0
HDFS-17315. Optimize the namenode format code logic. ( #6400 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-01-06 01:47:17 +09:00
LiuGuH
2369f0cddb
HDFS-17309. RBF: Fix Router Safemode check condition error ( #6390 ) Contributed by liuguanghua.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-05 18:36:28 +08:00
LiuGuH
7d3b6a36b8
HDFS-17306. RBF: Router should not return nameservices that does not enable observer nodes in RpcResponseHeaderProto ( #6385 )
2024-01-04 14:43:11 -08:00
hfutatzhanghb
d5468d84ba
HDFS-17283. Change the name of variable SECOND in HdfsClientConfigKeys. ( #6339 ). Contributed by farmmamba.
...
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-04 19:53:47 +08:00
huhaiyang
7a7db7f0dc
HDFS-17310. DiskBalancer: Enhance the log message for submitPlan ( #6391 ) Contributed by Haiyang Hu.
...
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-04 00:07:51 +08:00
LiuGuH
335587df9e
Add synchronized on lockLeakCheck() because threadCountMap is not thread safe. ( #6029 )
...
Co-authored-by: lgh <liuguanghua@kanzhun.com>
2024-01-03 21:12:38 +08:00
zzccctv
9f76fba6a4
Delete invalid code logic in namenode format ( #6323 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-29 22:25:05 +09:00
huangzhaobo
e26139beaa
HDFS-17301. Add read and write dataXceiver threads count metrics to datanode. ( #6377 )
...
Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-29 12:43:46 +09:00
huhaiyang
3aceec711b
HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted ( #6369 )
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-28 22:13:16 +09:00
zhtttylz
fc71dc3e94
HDFS-17284. Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery ( #6348 )
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-27 10:43:10 +09:00
Lei313
773dd7cc85
HDFS-17275: judge whether the block has been deleted in the block report ( #6335 ). Contributed by lei w.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2023-12-26 18:25:02 +08:00
huhaiyang
7e2ebfc17e
HDFS-17298. Fix NPE in DataNode.handleBadBlock and BlockSender ( #6374 )
2023-12-26 09:41:10 +09:00
xiaojunxiang
e07e445326
HDFS-17215. RBF: Fix some method annotations about @throws . ( #6136 ). Contributed by xiaojunxiang.
2023-12-26 00:37:58 +05:30
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. ( #6379 ). Contributed by huangzhaobo99.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-25 23:17:25 +05:30
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging ( #6321 ). Contributed by farmmamba.
...
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-24 16:57:04 +08:00
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java ( #6226 ). Contributed by Yu Wang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
2023-12-22 01:52:12 +05:30
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration ( #6347 ) Contributed by LiuGuH.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-12-21 12:21:59 +08:00
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. ( #6366 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-19 09:32:47 +09:00
