10033 Commits

Author SHA1 Message Date
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. (). Contributed by huangzhaobo99.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-25 23:17:25 +05:30
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging (). Contributed by farmmamba.
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-24 16:57:04 +08:00
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java (). Contributed by Yu Wang.
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
2023-12-22 01:52:12 +05:30
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration () Contributed by LiuGuH.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-12-21 12:21:59 +08:00
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. ()
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2023-12-19 09:32:47 +09:00
huangzhaobo
1498a8685d
HDFS-17282. Reconfig 'SlowIoWarningThreshold' parameters for datanode. (). Contributed by huangzhaobo99
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-14 11:47:53 +05:30
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. ()
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-12-13 20:56:53 +08:00
yijut2
81de229cf6
HDFS-17278. Fix order dependent flakiness in TestViewfsWithNfs3.java under hadoop-hdfs-nfs module (). Contributed by Ruby.
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-12 12:42:26 +05:30
caozhiqiang
37d6cada14
HDFS-17272. NNThroughputBenchmark should support specifying the base directory for multi-client test (). Contributed by caozhiqiang.
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-10 13:43:04 +05:30
huhaiyang
f96361fd54
HDFS-17279. RBF: Fix link to Fedbalance document (). Contributed by Haiyang Hu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-09 02:32:04 +05:30
Jian Zhang
29b0a6c28d
HDFS-17265. RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController () 2023-12-08 10:18:22 -08:00
Xing Lin
607c981042
HDFS-17262. Fixed the verbose log.warn in DFSUtil.addTransferRateMetric(). (). Contributed by Xing Lin.
Reviewed-by: Ravindra Dingankar <rdingankar@linkedin.com>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-07 10:16:23 +08:00
Thinker313
6a22bead54
HDFS-17270. RBF: Fix ZKDelegationTokenSecretManagerImpl use closed zookeeper client to get token in some case (). Contributed by lei w.
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-06 21:34:46 +08:00
huhaiyang
1fb80ef787
HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted (). Contributed by Haiyang Hu.
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-04 21:19:33 +08:00
huhaiyang
9a6d00aba4
HDFS-17218. NameNode should process time out excess redundancy blocks (). Contributed by Haiyang Hu.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-12-04 21:16:38 +08:00
huangzhaobo
2323ad24a2
HDFS-17260. Fix the logic for reconfigure slow peer enable for Namenode. (). Contributed by huangzhaobo99.
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-03 15:09:55 +05:30
Palakur Eshwitha Sai
d0b460f270
HDFS-17233. The conf dfs.datanode.lifeline.interval.seconds is not considering time unit seconds (). Contributed by Palakur Eshwitha Sai.
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-02 13:40:43 +05:30
Felix Nguyen
e76477e823
HDFS-17271. Fix dead DN sorting in web UI (). Contributed by Felix N.
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-02 13:37:22 +05:30
LiuGuH
071f850841
HDFS-17261. RBF: Fix getFileInfo return wrong path when get mountTable path which is multi-level (). Contributed by liuguanghua.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-12-01 17:05:58 +05:30
huangzhaobo
89e78a76a0
HDFS-17259. Fix typo in TestFsDatasetImpl Class. (). Contributed by huangzhaobo99. 2023-12-01 10:46:31 +05:30
LiuGuH
0417c1c633
HDFS-17263. RBF: Fix client ls trash path cannot get except default nameservices trash path () Contributed by liuguanghua.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-11-30 22:46:50 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (). Contributed by PJ Fanning.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-11-20 23:44:48 +05:30
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider () 2023-11-17 01:51:43 -08:00
QI Jiale
28068aa320
HDFS-17063. Support to configure different capacity reserved for each disk of DataNode. (). Contributed by QI Jiale.
Reviewed-by : Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-11-17 11:59:04 +08:00
huhaiyang
469f20a52b
HDFS-17172. Support FSNamesystemLock Parameters reconfigurable. (). Contributed by Haiyang Hu.
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-11-17 11:50:08 +08:00
Jian Zhang
513e6dcf14
HDFS-17232. RBF: Fix NoNamenodesAvailableException for a long time, when use observer. () 2023-11-16 09:13:48 -08:00
LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure ()
Contributed by liuguanghua.
2023-11-13 18:16:31 +00:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations ()
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
2023-11-08 14:25:21 +00:00
huhaiyang
4ef2322b6d
HDFS-17243. Add the parameter storage type for getBlocks method (). Contributed by Haiyang Hu.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2023-11-06 11:20:25 +08:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows () 2023-11-01 09:10:15 -07:00
PJ Fanning
a079f6261d
HADOOP-18917. Addendum. Fix deprecation issues after commons-io upgrade. (). Contributed by PJ Fanning. 2023-10-30 09:35:02 +05:30
Hiroaki Segawa
93a3c6e2cd
HDFS-17024. Potential data race introduced by HDFS-15865 () 2023-10-26 22:25:00 -07:00
Wei-Chiu Chuang
821ed83873
HDFS-15273. CacheReplicationMonitor hold lock for long time and lead to NN out of service. Contributed by Xiaoqiao He. 2023-10-26 10:35:10 -07:00
huhaiyang
f85ac5b60d
HADOOP-18920. RPC Metrics : Optimize logic for log slow RPCs () 2023-10-25 13:56:39 +08:00
gp1314
a170d58501
HDFS-17231. HA: Safemode should exit when resources are from low to available. (). Contributed by Gu Peng.
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-10-25 11:43:12 +08:00
Stephen O'Donnell
882f08b4bc
HDFS-17237. Remove IPCLoggerChannelMetrics when the logger is closed () 2023-10-24 21:39:03 +01:00
huhaiyang
9d48af8d70
HADOOP-18868. Optimize the configuration and use of callqueue overflow trigger failover () 2023-10-23 14:06:02 -07:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC () 2023-10-23 11:03:15 -07:00
huhaiyang
5eeab5e1b9
HDFS-17235. Fix javadoc errors in BlockManager (). Contributed by Haiyang Hu. 2023-10-23 20:12:39 +05:30
jianghuazhu
6e13e4addc
HDFS-17228. Improve documentation related to BlockManager. (). Contributed by JiangHua Zhu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-10-23 20:05:33 +05:30
Ayush Saxena
fbd653be9b
Revert "HDFS-17228. Improve documentation related to BlockManager. (). Contributed by JiangHua Zhu."
This reverts commit 81ba2e8484c4315bb9a765374df4bb2a05bc0ebd.
2023-10-23 19:35:12 +05:30
GuoPhilipse
615a2a42cf
HDFS-17220. Fix same available space policy in AvailableSpaceVolumeChoosingPolicy ()
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-10-18 13:01:32 +08:00
jianghuazhu
81ba2e8484
HDFS-17228. Improve documentation related to BlockManager. (). Contributed by JiangHua Zhu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-10-18 05:05:33 +05:30
GuoPhilipse
c8abca3004
HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. (). Contributed by GuoPhilipse.
Reviewed-by:  He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2023-10-16 16:34:40 +08:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. ()
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
2023-10-13 13:48:38 +01:00
huhaiyang
0ed484ac62
HDFS-17208. Add the metrics PendingAsyncDiskOperations in datanode (). Contributed by Haiyang Hu.
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-10-12 23:27:15 +08:00
Steve Vaughan
73eccd6d7c
HDFS-16740. Mini cluster test flakiness () 2023-10-10 13:51:46 -07:00
huhaiyang
85af6c3a28
HDFS-17217. Add lifeline RPC start up log when NameNode#startCommonServices (). Contributed by Haiyang Hu.
Reviewed-by:  Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2023-10-10 10:20:07 +08:00
hfutatzhanghb
ea3cb12ec8
HDFS-17171. CONGESTION_RATIO should be configurable ()
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-10-08 10:36:09 +08:00
huhaiyang
4c408a557f
HDFS-17205. HdfsServerConstants.MIN_BLOCKS_FOR_WRITE should be configurable (). Contributed by Haiyang Hu
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-10-06 12:39:23 +05:30