big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,232 Commits 2 Branches 2 Tags 264 MiB
8c378d1ea1
Commit Graph

9107 Commits

Author SHA1 Message Date
huhaiyang
3bd6a751ed
HDFS-17177. ErasureCodingWork should ignore the deleted block while reconstructing blocks (#6024) 2023-09-11 11:57:52 +08:00
dannytbecker
07c8b69f7c
HDFS-17167. Add config to startup NameNode as Observer (#6013) 2023-09-06 09:17:12 -07:00
Liangjun He
c2c6972f25
HDFS-17140. Revisit the BPOfferService.reportBadBlocks() method. (#5924). Contributed by Liangjun He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 20:19:04 +08:00
ZanderXu
7c941e00b4
HDFS-16933. A race in SerialNumberMap will cause wrong owner, group and XATTR. (#5430). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-06 10:27:52 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS (#5992) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-02 15:24:52 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
yuyanlei
b58885624b
HDFS-17093. Fix block report lease issue to avoid missing some storages report. (#5855). Contributed by Yanlei Yu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-28 14:52:38 +08:00
zhangshuyan
60867de422
HDFS-17151. EC: Fix wrong metadata in BlockInfoStriped after recovery. (#5938). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-21 10:05:34 +08:00
huhaiyang
65e4a66e25
HDFS-17087. Add Throttler for datanode reading block (#5845) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-17 09:33:50 +08:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. (#5941). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-16 20:29:19 +08:00
zhangshuyan
655c3df050
HDFS-17150. EC: Fix the bug of failed lease recovery. (#5937). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-15 19:54:15 +08:00
hfutatzhanghb
58314cbbf6
HDFS-17145. Fix description of property dfs.namenode.file.close.num-committed-allowed. (#5933) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-11 13:35:41 +08:00
huhaiyang
5b81caf0cf
HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication. (#5913). Contributed by Haiyang Hu. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-08 15:42:23 +08:00
hfutatzhanghb
b5b9a81538
HDFS-17144. Remove incorrect comment in method storeAllocatedBlock. (#5932) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-08-08 09:26:51 +08:00
Xing Lin
55e8301470
HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider (#5880) 2023-08-05 12:27:52 -07:00
zhtttylz
600dd45e47
HDFS-17122. Rectify the table length discrepancy in the DataNode UI (#5915) 2023-08-04 13:07:19 +08:00
WangYuanben
04c34a2171
HDFS-17113. Reconfig transfer and write bandwidth for datanode. (#5869) 2023-08-03 22:22:06 +08:00
zhiubok
52d66c2c11
HDFS-17136. Fix annotation description and typo in BlockPlacementPolicyDefault Class (#5907) 
Co-authored-by: huangzhaobo <huangzhaobo99@126.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:34:49 +08:00
huhaiyang
6cc56d40f5
HDFS-17135. Update fsck -blockId to display excess state info of blocks (#5904) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:21:22 +08:00
huhaiyang
1d09dcc614
HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. (#5877). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-27 11:28:30 +08:00
zhangshuyan
49c98da838
HDFS-17112. Show decommission duration in JMX and HTML. (#5866). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-24 14:38:00 +08:00
hfutatzhanghb
130bd033d5
HDFS-17074. Remove incorrect comment in TestRedudantBlocks#setup. (#5822). Contributed by farmmamba. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:45:40 +05:30
zhangshuyan
c35f31640e
HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. (#5847) 
Contributed by Shuyan Zhang
 2023-07-20 11:39:13 +01:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-20 09:21:45 +08:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) 2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) 2023-07-19 09:51:49 +08:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-16 11:27:31 +05:30
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 21:13:47 +05:30
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 00:58:43 +05:30
huhaiyang
4e699f0383
HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode (#5824). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 13:16:06 +05:30
WangYuanben
37b2d36edd
HDFS-17033. Update fsck to display stale state info of blocks accurately. (#5815). Contributed by Wang Yuanben. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-10 18:42:37 +08:00
huhaiyang
a84284e974
HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter (#5794). Contributed by Haiyang Hu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 13:08:47 +05:30
hfutatzhanghb
b673ebfec2
HDFS-17070. Remove unused import in DataNodeMetricHelper.java. (#5812). Contributed by farmmamba. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 12:58:23 +05:30
huhaiyang
3f983e49e0
HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" (#5817). Contributed by Haiyang Hu. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-09 23:04:20 +08:00
Zhaohui Wang
9659d0720d
HDFS-17065. Fix typos in hadoop-hdfs-project (#5796) 2023-07-03 08:41:37 -07:00
zhtttylz
750c0fc631
HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack. (#5759). Contributed by Hualong Zhang and Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-02 15:37:19 +08:00
wangzhaohui
8fb611c1f7
HDFS-17062. HA NameNode Web UI should show last HA Transition time (#5791). Contributed by Zhaohui Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-30 08:56:34 +05:30
huhaiyang
4bd873b816
HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. (#5735). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-28 12:03:15 +08:00
Xing Lin
03902f5ef0
HDFS-17055 Export HAState as a metric from Namenode for monitoring (#5764) 2023-06-26 15:52:59 -07:00
Wei-Chiu Chuang
e239d40ab1 Post release update 
* Add jdiff xml files from 3.3.6 release.
* Declare 3.3.6 as the latest stable release.
* Copy release notes.

(cherry picked from commit 7db9895000)
(cherry picked from commit cc121e2124aa01458dc296a060edc5e21a295268)
 2023-06-26 16:08:24 +00:00
hfutatzhanghb
e14c52c25f
HDFS-17053. Optimize method BlockInfoStriped#findSlot to reduce time complexity. (#5757). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-19 14:49:23 +05:30
wangzhaohui
02027c8dcc
HDFS-17051. Fix wrong time unit in TestFileAppend4#recoverFile (#5749). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 18:20:53 +05:30
Tsz-Wo Nicholas Sze
5dd37b2814
HDFS-17045. File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted. (#5738) 2023-06-15 03:30:04 +08:00
hfutatzhanghb
1a918a6b96
HDFS-17047. BlockManager#addStoredBlock should log storage id when AddBlockResult is REPLACED (#5742). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-15 00:24:22 +05:30
Xing Lin
ab47a658bd
HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy (#5700) 2023-06-14 10:53:56 -07:00
Dongjoon Hyun
fb16e00da0
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. 
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-11 11:38:13 +05:30
zhangshuyan
9c989515ba
HDFS-17037. Consider nonDfsUsed when running balancer. (#5715). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-09 16:19:08 +08:00
hfutatzhanghb
0e6bd09ae3
HDFS-17003. Erasure Coding: invalidate wrong block after reporting bad blocks from datanode (#5643). Contributed by hfutatzhanghb. 
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-08 18:06:51 +08:00
hfutatzhanghb
ddae78b0ec
HDFS-17035. FsVolumeImpl#getActualNonDfsUsed may return negative value. (#5708). Contributed by farmmamba. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-08 14:28:01 +05:30
huhaiyang
0c209961f8
HDFS-17019. Optimize the logic for reconfigure slow peer enable for Namenode" (#5671) 
* HDFS-17019. Optimize the logic for reconfigure slow peer enable for Namenode
 2023-06-08 10:05:49 +08:00
zhtttylz
d9980ab40f
HDFS-17029. Support getECPolices API in WebHDFS (#5698). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 17:33:37 +05:30
caozhiqiang
5d6ca13c5c
HDFS-16983. Fix concat operation doesn't honor dfs.permissions.enabled (#5561). Contributed by caozhiqiang. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 16:42:59 +05:30
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c.
 2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
NishthaShah
f8b7ddf69c
HDFS-16996. Fix flaky testFsCloseAfterClusterShutdown in TestFileCreation (#5697). Contributed by Nishtha Shah. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-01 08:23:48 +05:30
Marcono1234
9acf462d26
HDFS-17000. Fix faulty loop condition in TestDFSStripedOutputStreamUpdatePipeline (#5699). Contributed by Marcono1234. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-31 16:10:07 +05:30
hfutatzhanghb
e9740cb17a
HDFS-16908. Fix javadoc of field IncrementalBlockReportManager#readyToSend. (#5351). Contributed by farmmamba. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-23 16:12:50 +05:30
hfutatzhanghb
5b22dc6ace
HDFS-16909. Improve ReplicaMap#mergeAll method. (#5353). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-21 19:08:58 +08:00
huhaiyang
af933f3a4f
HDFS-17017. Fix the issue of arguments number limit in report command in DFSAdmin (#5667). Contributed by Haiyang Hu. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-21 09:03:34 +05:30
Keyao Li
0914b3e792
HDFS-16697. Add logs if resources are not available in NameNodeResourcePolicy. (#5569). Contributed by ECFuzz. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 18:38:01 +05:30
Keyao Li
339bc7b3a6
HDFS-16653. Improve error messages in ShortCircuitCache. (#5568). Contributed by ECFuzz. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 07:43:18 +05:30
LiuGuH
f6770dee47
HDFS-16979. RBF: Add proxyuser port in hdfsauditlog (#5552). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:02:16 +05:30
jianghuazhu
78cc528739
HDFS-17012.Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT. (#5659). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 14:27:19 +05:30
Tsz-Wo Nicholas Sze
350dcaf616
HDFS-17010. Add a subtree test to TestSnapshotDiffReport. (#5656) 2023-05-18 15:53:26 +08:00
wangzhaohui
03163f9de2
HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 12:12:11 +05:30
liang3zy22
482897a0f6
Fix typos in HDFS documents. (#5665) 2023-05-17 09:28:01 -07:00
smarthan
251439d769
HDFS-16985. Fix data missing issue when delete local block file. (#5564). Contributed by Chengwei Wang. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:33:38 +08:00
zhtttylz
0c77629849
HDFS-17001. Support getStatus API in WebHDFS (#5628). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 03:49:13 +05:30
slfan1989
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
zhangshuyan
03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). (#5622). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-09 10:05:28 +08:00
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
fanluoo
1a2cd965a7
HDFS-16897. Fix abundant Broken pipe exception in BlockSender (#5329). Contributed by fanluo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:24:56 +05:30
wangzhaohui
0e63152218
HDFS-16995. Remove unused parameters at NameNodeHttpServer#initWebHdfs (#5601). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-28 05:25:01 +05:30
Tsz-Wo Nicholas Sze
d9576bb9ee
HDFS-16972. Delete a snapshot may deleteCurrentFile. (#5532) 2023-04-27 09:17:47 -07:00
cxzl25
5af0845076
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-26 02:11:11 +05:30
zhtttylz
c9e0af9961
HDFS-16981. Support getFileLinkStatus API in WebHDFS (#5572). Contributed by Hualong Zhang. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 03:30:56 +05:30
Tsz-Wo Nicholas Sze
dc78849f27
HDFS-16975. FileWithSnapshotFeature.isCurrentFileDeleted is not reloaded from FSImage. (#5546) 2023-04-24 09:04:28 -07:00
zhangshuyan
6a23c376c9
HDFS-16986. EC: Fix locationBudget in getListing(). (#5582). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-24 18:53:25 +08:00
wangzhaohui
51dcbd1d61
HDFS-16988. Improve NameServices info at JournalNode web UI (#5584). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:13:02 +05:30
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
zhangshuyan
0185afafea
HDFS-16974. Consider volumes average load of each DataNode when choosing target. (#5541). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-14 10:33:30 +08:00
rdingankar
3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… (#5495) 2023-04-10 08:56:00 -07:00
zhtttylz
523ff81624
HDFS-16952. Support getLinkTarget API in WebHDFS (#5517) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-06 19:44:47 +08:00
zhtttylz
811441d5bc
HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc (#5491) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-01 18:18:20 +08:00
zhangshuyan
700147b4ac
HDFS-16964. Improve processing of excess redundancy after failover. (#5510). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-28 16:14:59 +08:00
Kidd5368
5cf62d1498
HDFS-16948. Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. (#5474). Contributed by Kidd53685368. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 17:00:23 +08:00
Ayush Saxena
e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Viraj Jasani
0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem (#5492) 2023-03-21 10:23:16 +08:00
Viraj Jasani
b6a9d7b442
HADOOP-18631. (ADDENDUM) Use LogCapturer to match audit log pattern and remove hdfs async audit log configs (#5451) 2023-03-18 06:33:50 +08:00
Stephen O'Donnell
eee2ea075d
HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/ 2023-03-15 10:03:00 +05:30
Stephen O'Donnell
ca6f5afb6d
HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460) 2023-03-11 16:40:07 +00:00
slfan1989
927401886a
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434) 
Contributed by Shilun Fan
 2023-03-06 15:26:53 +00:00
zhangshuyan
2cb0c35fc1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-06 20:10:31 +08:00
ZanderXu
6bd2444815
HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-03-01 16:18:38 -08:00
Tom
162288bc0a
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) 
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations. 
ignoredNodes list is only used on hedged read codepath

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-01 19:47:04 +00:00
Viraj Jasani
2ab7eb4caa
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:10 +00:00
rdingankar
0ca5686034
HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 18:26:32 +00:00
Viraj Jasani
a90238c0b8
HADOOP-18631. Migrate Async appenders to log4j properties (#5418) 2023-02-26 01:47:44 +08:00
Viraj Jasani
88914cada0
HDFS-16925. Namenode audit log to only include IP address of client (#5407) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:26:32 +08:00
hfutatzhanghb
723535b788
HDFS-16914. Add some logs for updateBlockForPipeline RPC. (#5381) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-16 09:25:50 +08:00
Zita Dombi
4cbe19f3a2
HDFS-16761. Namenode UI for Datanodes page not loading if any data node is down (#5390) 2023-02-15 16:16:04 +00:00
GuoPhilipse
fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Steve Vaughan
f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
Viraj Jasani
17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
huhaiyang
d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Steve Vaughan
08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao
3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao
7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb
eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
Viraj Jasani
f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
Viraj Jasani
bce388fd3f
HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 09:11:02 +08:00
Masatake Iwasaki
a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
huhaiyang
88c8ac750d
HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-31 17:46:38 +09:00
Szilard Nemeth
b677d40ab5 HADOOP-18602. Remove netty3 dependency 2023-01-27 16:32:50 +01:00
Viraj Jasani
04f3573f6a
HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-18 13:13:41 -08:00
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
Viraj Jasani
1263e024b9
HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 14:26:52 -08:00
huhaiyang
e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-01-11 09:12:26 +08:00
Chengbing Liu
4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-01-10 10:03:25 -08:00
huhaiyang
f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
Yubi Lee
4511c360b9
HDFS-16883. Duplicate field name in hdfs-default.xml (#5271). Contributed by YUBI LEE. 2023-01-05 04:58:43 +05:30
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
Daniel-009497
7ff326129d
HDFS-16871 DiskBalancer process may throw IllegalArgumentException when the target DataNode has capital letter in hostname (#5240) 2022-12-21 07:12:02 +08:00
Happy-shi
c5b42d59d2
HDFS-16866. Fix a typo in Dispatcher (#5202) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-16 11:07:41 +08:00
curie71
fdcbc8b072
HDFS-16868. Fix audit log duplicate issue when an ACE occurs in FSNamesystem. (#5206). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-12-13 12:24:51 +08:00
Anurag P
e76616f690
HDFS-16860 Upgrade moment.min.js to 2.29.4 (#5194) 2022-12-09 11:18:44 +05:30
dingshun3016
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. (#5180) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-09 08:10:04 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. (#5142) 2022-11-29 00:49:10 +00:00
ZanderXu
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md (#4922) 2022-11-25 09:26:28 +08:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
ZanderXu
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap (#5125) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:07:27 +08:00
huhaiyang
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB (#5129) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:00:27 +08:00
huhaiyang
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate (#5063) 2022-11-23 09:50:57 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
ZanderXu
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. (#4945). Contributed by ZanderXu. 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-11-13 22:01:35 +08:00
Takanobu Asanuma
660530205e
HDFS-16833. NameNode should log internal EC blocks instead of the EC block group when it receives block reports. (#5106) 
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-07 10:49:48 +09:00