big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,269 Commits 2 Branches 2 Tags 264 MiB
edf985e269
Commit Graph

10120 Commits

Author SHA1 Message Date
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes (#6508) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-18 09:15:40 +08:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up (#6498) 2024-02-13 16:24:23 -08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
 2024-02-13 16:33:51 +00:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider (#6510) 
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
 2024-02-13 10:49:39 +09:00
huhaiyang
5ad7737132
HDFS-17342. Fix DataNode may invalidates normal block causing missing block (#6464). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Chengwei Wang <1139557635@qq.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-02-06 17:52:52 +08:00
Takanobu Asanuma
9a7eeadaac
HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) 2024-02-06 11:54:14 +09:00
Alexander
20d8596af2
HDFS-17353. Fix failing RBF module tests. (#6491) Contributed by Alexander Bogdanov 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:26:30 +08:00
huhaiyang
6464507cd1
HDFS-17369. Add uuid into datanode info for NameNodeMXBean (#6521) Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:20:04 +08:00
hfutatzhanghb
4f4b846986
HDFS-17359. EC: recheck failed streamers should only after flushing all packets. (#6503). Contributed by farmmamba. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-02-01 23:53:37 +09:00
Lei313
ac471d7daa
HDFS-17339:Skip cacheReport when one blockPool does not have CacheBlock on this DataNode (#6456). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-25 21:15:20 +08:00
huhaiyang
caba9bbab3
HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt (#6476). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-25 12:49:18 +08:00
LiuGuH
8193a84020
HDFS-17324. RBF: Router should not return nameservices that not enable observer r… (#6412) 2024-01-23 12:30:08 -08:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. (#6368). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-22 11:50:51 +08:00
LiuGuH
2a1ee8dfcd
HDFS-17311. RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue. (#6392) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-20 07:55:23 +08:00
slfan1989
15e1789baf
Revert "HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998)" (#6457) Contributed by Shilun Fan. 
This reverts commit c1bf3cb0.

Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-20 07:51:55 +08:00
Jian Zhang
1036544480
HDFS-17302. RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation. (#6380) 2024-01-19 14:02:21 -08:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException (#6446) 
Print a warn log message for read retries and only print the full stack trace for a read request failure.

Contributed by: Xing Lin
 2024-01-18 18:03:28 -08:00
Lei313
cc4c4be1b7
HDFS-17331:Fix Blocks are always -1 and DataNode version are always UNKNOWN in federationhealth.html (#6429). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-18 21:10:54 +08:00
hfutatzhanghb
a30681077b
HDFS-17291. DataNode metric bytesWritten is not totally accurate in some situations. (#6360). Contributed by farmmamba. 
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:45:00 +08:00
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. (#6357). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:34:02 +08:00
hfutatzhanghb
6a053765ee
HDFS-17312. packetsReceived metric should ignore heartbeat packet. (#6394) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-11 22:08:37 +09:00
xuzifu666
99a59ae9e6
HDFS-17317. Improve the resource release for metaOut in DebugAdmin (#6402). Contributed by xy. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2024-01-07 00:59:31 +05:30
huangzhaobo
08713665c0
HDFS-17315. Optimize the namenode format code logic. (#6400) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-06 01:47:17 +09:00
LiuGuH
2369f0cddb
HDFS-17309. RBF: Fix Router Safemode check condition error (#6390) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-05 18:36:28 +08:00
LiuGuH
7d3b6a36b8
HDFS-17306. RBF: Router should not return nameservices that does not enable observer nodes in RpcResponseHeaderProto (#6385) 2024-01-04 14:43:11 -08:00
hfutatzhanghb
d5468d84ba
HDFS-17283. Change the name of variable SECOND in HdfsClientConfigKeys. (#6339). Contributed by farmmamba. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-04 19:53:47 +08:00
huhaiyang
7a7db7f0dc
HDFS-17310. DiskBalancer: Enhance the log message for submitPlan (#6391) Contributed by Haiyang Hu. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-04 00:07:51 +08:00
LiuGuH
335587df9e
Add synchronized on lockLeakCheck() because threadCountMap is not thread safe. (#6029) 
Co-authored-by: lgh <liuguanghua@kanzhun.com>
 2024-01-03 21:12:38 +08:00
zzccctv
9f76fba6a4
Delete invalid code logic in namenode format (#6323) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 22:25:05 +09:00
huangzhaobo
e26139beaa
HDFS-17301. Add read and write dataXceiver threads count metrics to datanode. (#6377) 
Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 12:43:46 +09:00
huhaiyang
3aceec711b
HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted (#6369) 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-28 22:13:16 +09:00
zhtttylz
fc71dc3e94
HDFS-17284. Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery (#6348) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-27 10:43:10 +09:00
Lei313
773dd7cc85
HDFS-17275: judge whether the block has been deleted in the block report (#6335). Contributed by lei w. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-26 18:25:02 +08:00
huhaiyang
7e2ebfc17e
HDFS-17298. Fix NPE in DataNode.handleBadBlock and BlockSender (#6374) 2023-12-26 09:41:10 +09:00
xiaojunxiang
e07e445326
HDFS-17215. RBF: Fix some method annotations about @throws . (#6136). Contributed by xiaojunxiang. 2023-12-26 00:37:58 +05:30
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. (#6379). Contributed by huangzhaobo99. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-25 23:17:25 +05:30
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging (#6321). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-24 16:57:04 +08:00
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java (#6226). Contributed by Yu Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2023-12-22 01:52:12 +05:30
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration (#6347) Contributed by LiuGuH. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-21 12:21:59 +08:00
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. (#6366) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-19 09:32:47 +09:00
huangzhaobo
1498a8685d
HDFS-17282. Reconfig 'SlowIoWarningThreshold' parameters for datanode. (#6338). Contributed by huangzhaobo99 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-14 11:47:53 +05:30
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. (#6227) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-12-13 20:56:53 +08:00
yijut2
81de229cf6
HDFS-17278. Fix order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module (#6329). Contributed by Ruby. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-12 12:42:26 +05:30
caozhiqiang
37d6cada14
HDFS-17272. NNThroughputBenchmark should support specifying the base directory for multi-client test (#6319). Contributed by caozhiqiang. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-10 13:43:04 +05:30
huhaiyang
f96361fd54
HDFS-17279. RBF: Fix link to Fedbalance document (#6333). Contributed by Haiyang Hu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-09 02:32:04 +05:30
Jian Zhang
29b0a6c28d
HDFS-17265. RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController (#6298) 2023-12-08 10:18:22 -08:00
Xing Lin
607c981042
HDFS-17262. Fixed the verbose log.warn in DFSUtil.addTransferRateMetric(). (#6290). Contributed by Xing Lin. 
Reviewed-by: Ravindra Dingankar <rdingankar@linkedin.com>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-07 10:16:23 +08:00
Thinker313
6a22bead54
HDFS-17270. RBF: Fix ZKDelegationTokenSecretManagerImpl use closed zookeeper client to get token in some case (#6315). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-06 21:34:46 +08:00
huhaiyang
1fb80ef787
HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted (#6266). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:19:33 +08:00
huhaiyang
9a6d00aba4
HDFS-17218. NameNode should process time out excess redundancy blocks (#6176). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:16:38 +08:00
huangzhaobo
2323ad24a2
HDFS-17260. Fix the logic for reconfigure slow peer enable for Namenode. (#6279). Contributed by huangzhaobo99. 
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-03 15:09:55 +05:30
Palakur Eshwitha Sai
d0b460f270
HDFS-17233. The conf dfs.datanode.lifeline.interval.seconds is not considering time unit seconds (#6286). Contributed by Palakur Eshwitha Sai. 
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-02 13:40:43 +05:30
Felix Nguyen
e76477e823
HDFS-17271. Fix dead DN sorting in web UI (#6318). Contributed by Felix N. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-02 13:37:22 +05:30
LiuGuH
071f850841
HDFS-17261. RBF: Fix getFileInfo return wrong path when get mountTable path which is multi-level (#6288). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-01 17:05:58 +05:30
huangzhaobo
89e78a76a0
HDFS-17259. Fix typo in TestFsDatasetImpl Class. (#6278). Contributed by huangzhaobo99. 2023-12-01 10:46:31 +05:30
LiuGuH
0417c1c633
HDFS-17263. RBF: Fix client ls trash path cannot get except default nameservices trash path (#6291) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-30 22:46:50 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider (#6263) 2023-11-17 01:51:43 -08:00
QI Jiale
28068aa320
HDFS-17063. Support to configure different capacity reserved for each disk of DataNode. (#5793). Contributed by QI Jiale. 
Reviewed-by : Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-11-17 11:59:04 +08:00
huhaiyang
469f20a52b
HDFS-17172. Support FSNamesystemLock Parameters reconfigurable. (#6002). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-11-17 11:50:08 +08:00
Jian Zhang
513e6dcf14
HDFS-17232. RBF: Fix NoNamenodesAvailableException for a long time, when use observer. (#6208) 2023-11-16 09:13:48 -08:00
LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure (#6249) 
Contributed by liuguanghua.
 2023-11-13 18:16:31 +00:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (#6198) 
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
 2023-11-08 14:25:21 +00:00
huhaiyang
4ef2322b6d
HDFS-17243. Add the parameter storage type for getBlocks method (#6238). Contributed by Haiyang Hu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-11-06 11:20:25 +08:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows (#5943) 2023-11-01 09:10:15 -07:00
PJ Fanning
a079f6261d
HADOOP-18917. Addendum. Fix deprecation issues after commons-io upgrade. (#6228). Contributed by PJ Fanning. 2023-10-30 09:35:02 +05:30
Hiroaki Segawa
93a3c6e2cd
HDFS-17024. Potential data race introduced by HDFS-15865 (#6223) 2023-10-26 22:25:00 -07:00
Wei-Chiu Chuang
821ed83873
HDFS-15273. CacheReplicationMonitor hold lock for long time and lead to NN out of service. Contributed by Xiaoqiao He. 2023-10-26 10:35:10 -07:00
huhaiyang
f85ac5b60d
HADOOP-18920. RPC Metrics : Optimize logic for log slow RPCs (#6146) 2023-10-25 13:56:39 +08:00
gp1314
a170d58501
HDFS-17231. HA: Safemode should exit when resources are from low to available. (#6207). Contributed by Gu Peng. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-10-25 11:43:12 +08:00
Stephen O'Donnell
882f08b4bc
HDFS-17237. Remove IPCLoggerChannelMetrics when the logger is closed (#6217) 2023-10-24 21:39:03 +01:00
huhaiyang
9d48af8d70
HADOOP-18868. Optimize the configuration and use of callqueue overflow trigger failover (#5998) 2023-10-23 14:06:02 -07:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
huhaiyang
5eeab5e1b9
HDFS-17235. Fix javadoc errors in BlockManager (#6214). Contributed by Haiyang Hu. 2023-10-23 20:12:39 +05:30
jianghuazhu
6e13e4addc
HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-23 20:05:33 +05:30
Ayush Saxena
fbd653be9b
Revert "HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu." 
This reverts commit 81ba2e8484.
 2023-10-23 19:35:12 +05:30
GuoPhilipse
615a2a42cf
HDFS-17220. Fix same available space policy in AvailableSpaceVolumeChoosingPolicy (#6174) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-10-18 13:01:32 +08:00
jianghuazhu
81ba2e8484
HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-18 05:05:33 +05:30
GuoPhilipse
c8abca3004
HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. (#6113). Contributed by GuoPhilipse. 
Reviewed-by:  He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-10-16 16:34:40 +08:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
huhaiyang
0ed484ac62
HDFS-17208. Add the metrics PendingAsyncDiskOperations in datanode (#6109). Contributed by Haiyang Hu. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-10-12 23:27:15 +08:00
Steve Vaughan
73eccd6d7c
HDFS-16740. Mini cluster test flakiness (#4835) 2023-10-10 13:51:46 -07:00
huhaiyang
85af6c3a28
HDFS-17217. Add lifeline RPC start up log when NameNode#startCommonServices (#6154). Contributed by Haiyang Hu. 
Reviewed-by:  Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-10-10 10:20:07 +08:00
hfutatzhanghb
ea3cb12ec8
HDFS-17171. CONGESTION_RATIO should be configurable (#5996) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-10-08 10:36:09 +08:00
huhaiyang
4c408a557f
HDFS-17205. HdfsServerConstants.MIN_BLOCKS_FOR_WRITE should be configurable (#6112). Contributed by Haiyang Hu 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-06 12:39:23 +05:30
huhaiyang
5edd21bc85
HDFS-17194. Enhance the log message for striped block recovery (#6094) 2023-10-04 11:22:59 -07:00
xiaojunxiang
0cfffb3012
HDFS-17214. RBF: The Quota class' andByStorageType method res has an incorrect initial value. (#6135) 
Co-authored-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
 2023-10-03 08:26:03 -07:00
Wang Yu
b87180568b
HDFS-17209. Correct comments to align with the code (#6110). Contributed by Yu Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-01 17:30:59 +05:30
ConfX
8931393302
HDFS-17133: TestFsDatasetImpl missing null check when cleaning up (#6079). Contributed by ConfX. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-09-29 12:06:24 +05:30
xiaojunxiang
390cd294f8
HDFS-17211. Fix comments in the RemoteParam class. (#6124). Contributed hellosrc. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-09-29 11:55:59 +05:30
zhangshuyan
26a5f38250
HDFS-17204. EC: Reduce unnecessary log when processing excess redundancy. (#6107). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-26 18:49:47 +08:00
zhangshuyan
ecee022e49
HDFS-17197. Show file replication when listing corrupt files. (#6095). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-25 13:01:25 +08:00
huhaiyang
cc66683b1a
HDFS-17184. Improve BlockReceiver to throws DiskOutOfSpaceException when initialize. (#6044). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-21 21:45:30 +08:00
Jian Zhang
d273c13ab5
HDFS-17198. RBF: fix bug of getRepresentativeQuorum when records have same dateModified (#6096) 2023-09-20 10:04:29 -07:00
ConfX
39f36d9071
HDFS-17105. mistakenly purge editLogs even after it is empty in NNStorageRetentionManager. (#6036). Contributed by ConfX. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-20 11:46:57 +08:00
PJ Fanning
c16484ffb2
HADOOP-18890. Remove use of okhttp in runtime code (#6057) 
Contributed by PJ Fanning
 2023-09-19 12:38:36 +01:00
章锡平
60f3a2b101
HDFS-17138 RBF: We changed the hadoop.security.auth_to_local configur… (#5921) 2023-09-18 09:40:22 -07:00
hfutatzhanghb
18f9989ff2
HDFS-17192. Add bock info when constructing remote block reader meets IOException. (#6081). Contributed by farmmamba. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-18 20:56:45 +08:00