Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 ( #5977 )
...
As well as the POM update, this patch moves to the (renamed) verify methods.
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.
Contributed by Anmol Asrani
2023-08-29 12:12:27 +01:00
yuyanlei
b58885624b
HDFS-17093. Fix block report lease issue to avoid missing some storages report. ( #5855 ). Contributed by Yanlei Yu.
...
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-28 14:52:38 +08:00
tian bao
11404c57cb
HDFS-17162. RBF: Add missing comments in StateStoreService #5959
...
Contributed by tian bao
2023-08-25 14:03:02 +01:00
Steve Loughran
2dfe947f20
Revert "Add missing comments in StateStoreService ( #5959 )"
...
This reverts commit 439c8653cf
2023-08-25 14:02:38 +01:00
tian bao
439c8653cf
Add missing comments in StateStoreService ( #5959 )
2023-08-23 18:30:09 -07:00
Benjamin Teke
43c889636a
YARN-11535. Remove jackson-dataformat-yaml dependency. ( #5970 )
2023-08-22 16:42:49 +02:00
zhangshuyan
60867de422
HDFS-17151. EC: Fix wrong metadata in BlockInfoStriped after recovery. ( #5938 ). Contributed by Shuyan Zhang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-21 10:05:34 +08:00
huhaiyang
65e4a66e25
HDFS-17087. Add Throttler for datanode reading block ( #5845 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-08-17 09:33:50 +08:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. ( #5941 ). Contributed by Shuyan Zhang.
...
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-16 20:29:19 +08:00
zhangshuyan
655c3df050
HDFS-17150. EC: Fix the bug of failed lease recovery. ( #5937 ). Contributed by Shuyan Zhang.
...
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-15 19:54:15 +08:00
hchaverri
ad2f45c64f
HDFS-17148. RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL ( #5936 )
2023-08-11 13:04:32 -07:00
hfutatzhanghb
58314cbbf6
HDFS-17145. Fix description of property dfs.namenode.file.close.num-committed-allowed. ( #5933 )
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-08-11 13:35:41 +08:00
huhaiyang
5b81caf0cf
HDFS-17137. Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication. ( #5913 ). Contributed by Haiyang Hu.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-08 15:42:23 +08:00
hfutatzhanghb
b5b9a81538
HDFS-17144. Remove incorrect comment in method storeAllocatedBlock. ( #5932 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2023-08-08 09:26:51 +08:00
hchaverri
bc48e5cbe8
HDFS-17128. Updating SQLDelegationTokenSecretManager to use LoadingCache so tokens are updated frequently. ( #5897 ) Contributed by Hector Sandoval Chaverri.
...
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-08-08 07:45:14 +08:00
Xing Lin
55e8301470
HDFS-17118 Fixed a couple checkstyle warnings in TestObserverReadProxyProvider ( #5880 )
2023-08-05 12:27:52 -07:00
zhtttylz
600dd45e47
HDFS-17122. Rectify the table length discrepancy in the DataNode UI ( #5915 )
2023-08-04 13:07:19 +08:00
WangYuanben
04c34a2171
HDFS-17113. Reconfig transfer and write bandwidth for datanode. ( #5869 )
2023-08-03 22:22:06 +08:00
zhangshuyan
5aee0e0c0a
HDFS-17134. RBF: Fix duplicate results of getListing through Router. ( #5900 ). Contributed by Shuyan Zhang.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-01 17:52:54 +08:00
zhiubok
52d66c2c11
HDFS-17136. Fix annotation description and typo in BlockPlacementPolicyDefault Class ( #5907 )
...
Co-authored-by: huangzhaobo <huangzhaobo99@126.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-07-31 09:34:49 +08:00
huhaiyang
6cc56d40f5
HDFS-17135. Update fsck -blockId to display excess state info of blocks ( #5904 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-07-31 09:21:22 +08:00
huhaiyang
87c036efb3
HDFS-17116. RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService ( #5876 ). Contributed by Haiyang Hu.
...
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-28 15:15:00 +08:00
huhaiyang
1d09dcc614
HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. ( #5877 ). Contributed by Haiyang Hu.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-27 11:28:30 +08:00
zhtttylz
068d8c7e4d
HDFS-17115. HttpFS Add Support getErasureCodeCodecs API ( #5875 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-26 17:40:08 +05:30
Simbarashe Dzinamarira
ad001c93cf
HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled. ( #5860 ). Contributed by Simbarashe Dzinamarira.
2023-07-26 13:22:13 +05:30
Zhaohui Wang
9cfe9ccd26
HDFS-17119. RBF: Logger fix for StateStoreMySQLImpl. ( #5882 ). Contributed by Zhaohui Wang.
...
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-24 17:26:23 +05:30
zhangshuyan
49c98da838
HDFS-17112. Show decommission duration in JMX and HTML. ( #5866 ). Contributed by Shuyan Zhang.
...
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-24 14:38:00 +08:00
hfutatzhanghb
130bd033d5
HDFS-17074. Remove incorrect comment in TestRedudantBlocks#setup. ( #5822 ). Contributed by farmmamba.
...
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-21 00:45:40 +05:30
Xing Lin
80fefd093f
HDFS-17067 Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy ( #5803 )
2023-07-20 10:46:41 -07:00
zhangshuyan
c35f31640e
HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. ( #5847 )
...
Contributed by Shuyan Zhang
2023-07-20 11:39:13 +01:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. ( #5854 )
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-07-20 09:21:45 +08:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md ( #5846 )
2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. ( #5849 )
2023-07-19 09:51:49 +08:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode ( #5823 ). Contributed by Haiyang Hu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-16 11:27:31 +05:30
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota ( #5842 ). Contributed by Haiyang Hu.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. ( #5809 ). Contributed by farmmamba.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-14 21:13:47 +05:30
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated ( #5833 ). Contributed by Haiyang Hu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS ( #5836 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" ( #5834 ). Contributed by Haiyang Hu.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-14 00:58:43 +05:30
huhaiyang
4e699f0383
HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode ( #5824 ). Contributed by Haiyang Hu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-11 13:16:06 +05:30
WangYuanben
37b2d36edd
HDFS-17033. Update fsck to display stale state info of blocks accurately. ( #5815 ). Contributed by Wang Yuanben.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-10 18:42:37 +08:00
huhaiyang
a84284e974
HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter ( #5794 ). Contributed by Haiyang Hu.
...
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-10 13:08:47 +05:30
hfutatzhanghb
b673ebfec2
HDFS-17070. Remove unused import in DataNodeMetricHelper.java. ( #5812 ). Contributed by farmmamba.
...
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-10 12:58:23 +05:30
huhaiyang
3f983e49e0
HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" ( #5817 ). Contributed by Haiyang Hu.
...
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-09 23:04:20 +08:00
Zhaohui Wang
9659d0720d
HDFS-17065. Fix typos in hadoop-hdfs-project ( #5796 )
2023-07-03 08:41:37 -07:00
zhtttylz
750c0fc631
HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack. ( #5759 ). Contributed by Hualong Zhang and Shuyan Zhang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-07-02 15:37:19 +08:00
wangzhaohui
8fb611c1f7
HDFS-17062. HA NameNode Web UI should show last HA Transition time ( #5791 ). Contributed by Zhaohui Wang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-06-30 08:56:34 +05:30
huhaiyang
4bd873b816
HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. ( #5735 ). Contributed by Haiyang Hu.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-06-28 12:03:15 +08:00
Xing Lin
03902f5ef0
HDFS-17055 Export HAState as a metric from Namenode for monitoring ( #5764 )
2023-06-26 15:52:59 -07:00
huhaiyang
161f80810f
HDFS-17057. Add DataNode maintenance states to Federation UI ( #5775 )
2023-06-26 09:38:57 -07:00
