big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,367 Commits 2 Branches 2 Tags 264 MiB
2cf4d638af
Commit Graph

9142 Commits

Author SHA1 Message Date
slfan1989
ff3f2255d2
HADOOP-19112. Hadoop 3.4.0 release wrap-up. (#6640) Contributed by Shilun Fan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-19 20:08:03 +08:00
Tsz-Wo Nicholas Sze
b25b28e5bb
HDFS-17380. FsImageValidation: remove inaccessible nodes. (#6549). Contributed by Tsz-wo Sze. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-18 11:52:17 +08:00
Lei313
e211f6f83d
HDFS-17391. Adjust the checkpoint io buffer size to the chunk size (#6594). Contributed by lei w. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-12 18:36:43 +08:00
Lei313
dbf08c872a
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-12 12:25:33 +08:00
ritegarg
58afe43769
HDFS-17299. Adding rack failure tolerance when creating a new file (#6566) 2024-03-06 13:08:05 -08:00
hfutatzhanghb
7012986fc3
HDFS-17345. Add a metrics to record block report generating cost time. (#6475). Contributed by farmmamba. 
Reviewed-by:  Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-03-06 16:59:00 +08:00
dannytbecker
5584efd8d4
HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade (#6583) 2024-03-05 11:31:31 -08:00
Jtdellaringa
a2d7241190
HDFS-17404. Add NN Socket Address to log when processing command from active NN (#6598) 
* Add nnSocketAddress to log when processing command from active nn
 2024-03-05 10:33:22 -08:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. (#6430) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-03-02 21:35:24 +09:00
hfutatzhanghb
15af52954f
HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. (#6509). Contributed by farmmamba. 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-02-27 10:19:57 +08:00
ZanderXu
a897e745f5
HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem (#6567) 2024-02-23 15:37:27 +08:00
Lars Francke
d1daf26b85
Fixes HDFS-17181 by routing all CREATE requests to the BlockManager (#6108) 2024-02-20 10:58:49 -08:00
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes (#6508) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-18 09:15:40 +08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
 2024-02-13 16:33:51 +00:00
huhaiyang
5ad7737132
HDFS-17342. Fix DataNode may invalidates normal block causing missing block (#6464). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Chengwei Wang <1139557635@qq.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-02-06 17:52:52 +08:00
huhaiyang
6464507cd1
HDFS-17369. Add uuid into datanode info for NameNodeMXBean (#6521) Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:20:04 +08:00
Lei313
ac471d7daa
HDFS-17339:Skip cacheReport when one blockPool does not have CacheBlock on this DataNode (#6456). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-25 21:15:20 +08:00
huhaiyang
caba9bbab3
HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt (#6476). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-25 12:49:18 +08:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. (#6368). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-22 11:50:51 +08:00
slfan1989
15e1789baf
Revert "HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998)" (#6457) Contributed by Shilun Fan. 
This reverts commit c1bf3cb0.

Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-20 07:51:55 +08:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException (#6446) 
Print a warn log message for read retries and only print the full stack trace for a read request failure.

Contributed by: Xing Lin
 2024-01-18 18:03:28 -08:00
hfutatzhanghb
a30681077b
HDFS-17291. DataNode metric bytesWritten is not totally accurate in some situations. (#6360). Contributed by farmmamba. 
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:45:00 +08:00
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. (#6357). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:34:02 +08:00
hfutatzhanghb
6a053765ee
HDFS-17312. packetsReceived metric should ignore heartbeat packet. (#6394) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-11 22:08:37 +09:00
xuzifu666
99a59ae9e6
HDFS-17317. Improve the resource release for metaOut in DebugAdmin (#6402). Contributed by xy. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2024-01-07 00:59:31 +05:30
huangzhaobo
08713665c0
HDFS-17315. Optimize the namenode format code logic. (#6400) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-01-06 01:47:17 +09:00
huhaiyang
7a7db7f0dc
HDFS-17310. DiskBalancer: Enhance the log message for submitPlan (#6391) Contributed by Haiyang Hu. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-04 00:07:51 +08:00
LiuGuH
335587df9e
Add synchronized on lockLeakCheck() because threadCountMap is not thread safe. (#6029) 
Co-authored-by: lgh <liuguanghua@kanzhun.com>
 2024-01-03 21:12:38 +08:00
zzccctv
9f76fba6a4
Delete invalid code logic in namenode format (#6323) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 22:25:05 +09:00
huangzhaobo
e26139beaa
HDFS-17301. Add read and write dataXceiver threads count metrics to datanode. (#6377) 
Reviewed-by: hfutatzhanghb <hfutzhanghb@163.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-29 12:43:46 +09:00
huhaiyang
3aceec711b
HDFS-17297. The NameNode should remove block from the BlocksMap if the block is marked as deleted (#6369) 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-28 22:13:16 +09:00
zhtttylz
fc71dc3e94
HDFS-17284. Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery (#6348) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-27 10:43:10 +09:00
Lei313
773dd7cc85
HDFS-17275: judge whether the block has been deleted in the block report (#6335). Contributed by lei w. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-12-26 18:25:02 +08:00
huhaiyang
7e2ebfc17e
HDFS-17298. Fix NPE in DataNode.handleBadBlock and BlockSender (#6374) 2023-12-26 09:41:10 +09:00
huangzhaobo
630ffb280d
HDFS-17056. EC: Fix verifyClusterSetup output in case of an invalid param. (#6379). Contributed by huangzhaobo99. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-25 23:17:25 +05:30
Wang Yu
77edca8f0a
HDFS-17240. Fix a typo in DataStorage.java (#6226). Contributed by Yu Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2023-12-22 01:52:12 +05:30
huangzhaobo
95ea31fafb
HDFS-17294. Reconfigure the scheduling cycle of the slowPeerCollectorDaemon thread. (#6366) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-12-19 09:32:47 +09:00
huangzhaobo
1498a8685d
HDFS-17282. Reconfig 'SlowIoWarningThreshold' parameters for datanode. (#6338). Contributed by huangzhaobo99 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-14 11:47:53 +05:30
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. (#6227) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-12-13 20:56:53 +08:00
caozhiqiang
37d6cada14
HDFS-17272. NNThroughputBenchmark should support specifying the base directory for multi-client test (#6319). Contributed by caozhiqiang. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-10 13:43:04 +05:30
Xing Lin
607c981042
HDFS-17262. Fixed the verbose log.warn in DFSUtil.addTransferRateMetric(). (#6290). Contributed by Xing Lin. 
Reviewed-by: Ravindra Dingankar <rdingankar@linkedin.com>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-07 10:16:23 +08:00
huhaiyang
1fb80ef787
HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted (#6266). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:19:33 +08:00
huhaiyang
9a6d00aba4
HDFS-17218. NameNode should process time out excess redundancy blocks (#6176). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-04 21:16:38 +08:00
huangzhaobo
2323ad24a2
HDFS-17260. Fix the logic for reconfigure slow peer enable for Namenode. (#6279). Contributed by huangzhaobo99. 
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-03 15:09:55 +05:30
Palakur Eshwitha Sai
d0b460f270
HDFS-17233. The conf dfs.datanode.lifeline.interval.seconds is not considering time unit seconds (#6286). Contributed by Palakur Eshwitha Sai. 
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-02 13:40:43 +05:30
Felix Nguyen
e76477e823
HDFS-17271. Fix dead DN sorting in web UI (#6318). Contributed by Felix N. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-02 13:37:22 +05:30
huangzhaobo
89e78a76a0
HDFS-17259. Fix typo in TestFsDatasetImpl Class. (#6278). Contributed by huangzhaobo99. 2023-12-01 10:46:31 +05:30
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider (#6263) 2023-11-17 01:51:43 -08:00
QI Jiale
28068aa320
HDFS-17063. Support to configure different capacity reserved for each disk of DataNode. (#5793). Contributed by QI Jiale. 
Reviewed-by : Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-11-17 11:59:04 +08:00
huhaiyang
469f20a52b
HDFS-17172. Support FSNamesystemLock Parameters reconfigurable. (#6002). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-11-17 11:50:08 +08:00
LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure (#6249) 
Contributed by liuguanghua.
 2023-11-13 18:16:31 +00:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (#6198) 
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
 2023-11-08 14:25:21 +00:00
huhaiyang
4ef2322b6d
HDFS-17243. Add the parameter storage type for getBlocks method (#6238). Contributed by Haiyang Hu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-11-06 11:20:25 +08:00
PJ Fanning
a079f6261d
HADOOP-18917. Addendum. Fix deprecation issues after commons-io upgrade. (#6228). Contributed by PJ Fanning. 2023-10-30 09:35:02 +05:30
Wei-Chiu Chuang
821ed83873
HDFS-15273. CacheReplicationMonitor hold lock for long time and lead to NN out of service. Contributed by Xiaoqiao He. 2023-10-26 10:35:10 -07:00
huhaiyang
f85ac5b60d
HADOOP-18920. RPC Metrics : Optimize logic for log slow RPCs (#6146) 2023-10-25 13:56:39 +08:00
gp1314
a170d58501
HDFS-17231. HA: Safemode should exit when resources are from low to available. (#6207). Contributed by Gu Peng. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-10-25 11:43:12 +08:00
Stephen O'Donnell
882f08b4bc
HDFS-17237. Remove IPCLoggerChannelMetrics when the logger is closed (#6217) 2023-10-24 21:39:03 +01:00
huhaiyang
9d48af8d70
HADOOP-18868. Optimize the configuration and use of callqueue overflow trigger failover (#5998) 2023-10-23 14:06:02 -07:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
huhaiyang
5eeab5e1b9
HDFS-17235. Fix javadoc errors in BlockManager (#6214). Contributed by Haiyang Hu. 2023-10-23 20:12:39 +05:30
jianghuazhu
6e13e4addc
HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-23 20:05:33 +05:30
Ayush Saxena
fbd653be9b
Revert "HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu." 
This reverts commit 81ba2e8484.
 2023-10-23 19:35:12 +05:30
GuoPhilipse
615a2a42cf
HDFS-17220. Fix same available space policy in AvailableSpaceVolumeChoosingPolicy (#6174) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-10-18 13:01:32 +08:00
jianghuazhu
81ba2e8484
HDFS-17228. Improve documentation related to BlockManager. (#6195). Contributed by JiangHua Zhu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-18 05:05:33 +05:30
GuoPhilipse
c8abca3004
HDFS-17210. Optimize AvailableSpaceBlockPlacementPolicy. (#6113). Contributed by GuoPhilipse. 
Reviewed-by:  He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-10-16 16:34:40 +08:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
huhaiyang
0ed484ac62
HDFS-17208. Add the metrics PendingAsyncDiskOperations in datanode (#6109). Contributed by Haiyang Hu. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-10-12 23:27:15 +08:00
Steve Vaughan
73eccd6d7c
HDFS-16740. Mini cluster test flakiness (#4835) 2023-10-10 13:51:46 -07:00
huhaiyang
85af6c3a28
HDFS-17217. Add lifeline RPC start up log when NameNode#startCommonServices (#6154). Contributed by Haiyang Hu. 
Reviewed-by:  Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-10-10 10:20:07 +08:00
hfutatzhanghb
ea3cb12ec8
HDFS-17171. CONGESTION_RATIO should be configurable (#5996) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-10-08 10:36:09 +08:00
huhaiyang
4c408a557f
HDFS-17205. HdfsServerConstants.MIN_BLOCKS_FOR_WRITE should be configurable (#6112). Contributed by Haiyang Hu 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-06 12:39:23 +05:30
huhaiyang
5edd21bc85
HDFS-17194. Enhance the log message for striped block recovery (#6094) 2023-10-04 11:22:59 -07:00
Wang Yu
b87180568b
HDFS-17209. Correct comments to align with the code (#6110). Contributed by Yu Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-10-01 17:30:59 +05:30
ConfX
8931393302
HDFS-17133: TestFsDatasetImpl missing null check when cleaning up (#6079). Contributed by ConfX. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-09-29 12:06:24 +05:30
zhangshuyan
26a5f38250
HDFS-17204. EC: Reduce unnecessary log when processing excess redundancy. (#6107). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-26 18:49:47 +08:00
zhangshuyan
ecee022e49
HDFS-17197. Show file replication when listing corrupt files. (#6095). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-25 13:01:25 +08:00
huhaiyang
cc66683b1a
HDFS-17184. Improve BlockReceiver to throws DiskOutOfSpaceException when initialize. (#6044). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-21 21:45:30 +08:00
ConfX
39f36d9071
HDFS-17105. mistakenly purge editLogs even after it is empty in NNStorageRetentionManager. (#6036). Contributed by ConfX. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-20 11:46:57 +08:00
章锡平
60f3a2b101
HDFS-17138 RBF: We changed the hadoop.security.auth_to_local configur… (#5921) 2023-09-18 09:40:22 -07:00
zhangshuyan
9e489b9ab5
HDFS-17190. EC: Fix bug of OIV processing XAttr. (#6067). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-18 10:35:13 +08:00
dannytbecker
4652d22b91
HDFS-17178: BootstrapStandby needs to handle RollingUpgrade (#6018) 2023-09-12 14:46:08 -07:00
huhaiyang
3bd6a751ed
HDFS-17177. ErasureCodingWork should ignore the deleted block while reconstructing blocks (#6024) 2023-09-11 11:57:52 +08:00
dannytbecker
07c8b69f7c
HDFS-17167. Add config to startup NameNode as Observer (#6013) 2023-09-06 09:17:12 -07:00
Liangjun He
c2c6972f25
HDFS-17140. Revisit the BPOfferService.reportBadBlocks() method. (#5924). Contributed by Liangjun He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 20:19:04 +08:00
ZanderXu
7c941e00b4
HDFS-16933. A race in SerialNumberMap will cause wrong owner, group and XATTR. (#5430). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 10:27:52 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS (#5992) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-02 15:24:52 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
yuyanlei
b58885624b
HDFS-17093. Fix block report lease issue to avoid missing some storages report. (#5855). Contributed by Yanlei Yu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-28 14:52:38 +08:00
zhangshuyan
60867de422
HDFS-17151. EC: Fix wrong metadata in BlockInfoStriped after recovery. (#5938). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-21 10:05:34 +08:00
huhaiyang
65e4a66e25
HDFS-17087. Add Throttler for datanode reading block (#5845) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-17 09:33:50 +08:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. (#5941). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-16 20:29:19 +08:00
zhangshuyan
655c3df050
HDFS-17150. EC: Fix the bug of failed lease recovery. (#5937). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-15 19:54:15 +08:00
hfutatzhanghb
58314cbbf6
HDFS-17145. Fix description of property dfs.namenode.file.close.num-committed-allowed. (#5933) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-11 13:35:41 +08:00
huhaiyang
5b81caf0cf
HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication. (#5913). Contributed by Haiyang Hu. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-08 15:42:23 +08:00
hfutatzhanghb
b5b9a81538
HDFS-17144. Remove incorrect comment in method storeAllocatedBlock. (#5932) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-08 09:26:51 +08:00