big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,853 Commits 2 Branches 2 Tags 264 MiB
07c8b69f7c
Commit Graph

9006 Commits

Author SHA1 Message Date
dannytbecker
07c8b69f7c
HDFS-17167. Add config to startup NameNode as Observer (#6013) 2023-09-06 09:17:12 -07:00
Liangjun He
c2c6972f25
HDFS-17140. Revisit the BPOfferService.reportBadBlocks() method. (#5924). Contributed by Liangjun He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 20:19:04 +08:00
ZanderXu
7c941e00b4
HDFS-16933. A race in SerialNumberMap will cause wrong owner, group and XATTR. (#5430). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 10:27:52 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS (#5992) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-02 15:24:52 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
yuyanlei
b58885624b
HDFS-17093. Fix block report lease issue to avoid missing some storages report. (#5855). Contributed by Yanlei Yu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-28 14:52:38 +08:00
zhangshuyan
60867de422
HDFS-17151. EC: Fix wrong metadata in BlockInfoStriped after recovery. (#5938). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-21 10:05:34 +08:00
huhaiyang
65e4a66e25
HDFS-17087. Add Throttler for datanode reading block (#5845) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-17 09:33:50 +08:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. (#5941). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-16 20:29:19 +08:00
zhangshuyan
655c3df050
HDFS-17150. EC: Fix the bug of failed lease recovery. (#5937). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-15 19:54:15 +08:00
hfutatzhanghb
58314cbbf6
HDFS-17145. Fix description of property dfs.namenode.file.close.num-committed-allowed. (#5933) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-11 13:35:41 +08:00
huhaiyang
5b81caf0cf
HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication. (#5913). Contributed by Haiyang Hu. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-08 15:42:23 +08:00
hfutatzhanghb
b5b9a81538
HDFS-17144. Remove incorrect comment in method storeAllocatedBlock. (#5932) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-08 09:26:51 +08:00
Xing Lin
55e8301470
HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider (#5880) 2023-08-05 12:27:52 -07:00
zhtttylz
600dd45e47
HDFS-17122. Rectify the table length discrepancy in the DataNode UI (#5915) 2023-08-04 13:07:19 +08:00
WangYuanben
04c34a2171
HDFS-17113. Reconfig transfer and write bandwidth for datanode. (#5869) 2023-08-03 22:22:06 +08:00
zhiubok
52d66c2c11
HDFS-17136. Fix annotation description and typo in BlockPlacementPolicyDefault Class (#5907) 
Co-authored-by: huangzhaobo <huangzhaobo99@126.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:34:49 +08:00
huhaiyang
6cc56d40f5
HDFS-17135. Update fsck -blockId to display excess state info of blocks (#5904) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:21:22 +08:00
huhaiyang
1d09dcc614
HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. (#5877). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-27 11:28:30 +08:00
zhangshuyan
49c98da838
HDFS-17112. Show decommission duration in JMX and HTML. (#5866). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-24 14:38:00 +08:00
hfutatzhanghb
130bd033d5
HDFS-17074. Remove incorrect comment in TestRedudantBlocks#setup. (#5822). Contributed by farmmamba. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:45:40 +05:30
zhangshuyan
c35f31640e
HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. (#5847) 
Contributed by Shuyan Zhang
 2023-07-20 11:39:13 +01:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-20 09:21:45 +08:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) 2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) 2023-07-19 09:51:49 +08:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-16 11:27:31 +05:30
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 21:13:47 +05:30
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 00:58:43 +05:30
huhaiyang
4e699f0383
HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode (#5824). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 13:16:06 +05:30
WangYuanben
37b2d36edd
HDFS-17033. Update fsck to display stale state info of blocks accurately. (#5815). Contributed by Wang Yuanben. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-10 18:42:37 +08:00
huhaiyang
a84284e974
HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter (#5794). Contributed by Haiyang Hu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 13:08:47 +05:30
hfutatzhanghb
b673ebfec2
HDFS-17070. Remove unused import in DataNodeMetricHelper.java. (#5812). Contributed by farmmamba. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 12:58:23 +05:30
huhaiyang
3f983e49e0
HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" (#5817). Contributed by Haiyang Hu. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-09 23:04:20 +08:00
Zhaohui Wang
9659d0720d
HDFS-17065. Fix typos in hadoop-hdfs-project (#5796) 2023-07-03 08:41:37 -07:00
zhtttylz
750c0fc631
HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack. (#5759). Contributed by Hualong Zhang and Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-02 15:37:19 +08:00
wangzhaohui
8fb611c1f7
HDFS-17062. HA NameNode Web UI should show last HA Transition time (#5791). Contributed by Zhaohui Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-30 08:56:34 +05:30
huhaiyang
4bd873b816
HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. (#5735). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-28 12:03:15 +08:00
Xing Lin
03902f5ef0
HDFS-17055 Export HAState as a metric from Namenode for monitoring (#5764) 2023-06-26 15:52:59 -07:00
Wei-Chiu Chuang
e239d40ab1 Post release update 
* Add jdiff xml files from 3.3.6 release.
* Declare 3.3.6 as the latest stable release.
* Copy release notes.

(cherry picked from commit 7db9895000)
(cherry picked from commit cc121e2124aa01458dc296a060edc5e21a295268)
 2023-06-26 16:08:24 +00:00
hfutatzhanghb
e14c52c25f
HDFS-17053. Optimize method BlockInfoStriped#findSlot to reduce time complexity. (#5757). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-19 14:49:23 +05:30
wangzhaohui
02027c8dcc
HDFS-17051. Fix wrong time unit in TestFileAppend4#recoverFile (#5749). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 18:20:53 +05:30
Tsz-Wo Nicholas Sze
5dd37b2814
HDFS-17045. File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted. (#5738) 2023-06-15 03:30:04 +08:00
hfutatzhanghb
1a918a6b96
HDFS-17047. BlockManager#addStoredBlock should log storage id when AddBlockResult is REPLACED (#5742). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 00:24:22 +05:30
Xing Lin
ab47a658bd
HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy (#5700) 2023-06-14 10:53:56 -07:00
Dongjoon Hyun
fb16e00da0
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. 
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-11 11:38:13 +05:30
zhangshuyan
9c989515ba
HDFS-17037. Consider nonDfsUsed when running balancer. (#5715). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-09 16:19:08 +08:00