big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,084 Commits 2 Branches 2 Tags 264 MiB
bc159b5a87
Commit Graph

27084 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sadanand Shenoy
b971222372
HDFS-17120. Support snapshot diff based copylisting for flat paths. (#5885) 2023-07-27 00:53:57 -07:00
Benjamin Teke
b1fc00d4b2
YARN-11539. Fix leaf-templates in Flexible AQC. (#5868) 2023-07-27 09:24:35 +02:00
huhaiyang
1d09dcc614
HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. (#5877). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-27 11:28:30 +08:00
ahmarsuhail
24f5f708df
HADOOP-18778. Fixes failing tests when CSE is enabled. (#5763) 
Contributed By: Ahmar Suhail <ahmarsu@amazon.co.uk>
 2023-07-26 21:56:49 +05:30
zhtttylz
068d8c7e4d
HDFS-17115. HttpFS Add Support getErasureCodeCodecs API (#5875). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-26 17:40:08 +05:30
Simbarashe Dzinamarira
ad001c93cf
HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled. (#5860). Contributed by Simbarashe Dzinamarira. 2023-07-26 13:22:13 +05:30
Ayush Saxena
c04a17f116
HADOOP-18823. Add Labeler Github Action. (#5874). Contributed by Ayush Saxena. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-25 03:04:49 +05:30
Viraj Jasani
90793e1bce
HADOOP-18805. S3A prefetch tests to work with small files (#5851) 
Contributed by Viraj Jasani
 2023-07-24 19:36:57 +01:00
yu liang
caf1816e0e
MAPREDUCE-7442. Exception message is not intusive when accessing the job configuration web UI (#5848) 2023-07-24 21:40:36 +08:00
Zhaohui Wang
9cfe9ccd26
HDFS-17119. RBF: Logger fix for StateStoreMySQLImpl. (#5882). Contributed by Zhaohui Wang. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-24 17:26:23 +05:30
zhangshuyan
49c98da838
HDFS-17112. Show decommission duration in JMX and HTML. (#5866). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-24 14:38:00 +08:00
Peter Szucs
4d4b099309
YARN-11534. Fixed exception handling when container signalling is interrupted (#5864) 2023-07-21 12:30:55 +02:00
hfutatzhanghb
130bd033d5
HDFS-17074. Remove incorrect comment in TestRedudantBlocks#setup. (#5822). Contributed by farmmamba. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:45:40 +05:30
yoonsch217
fbe9a29246
YARN-11540. Fix typo: form -> from (#5861). Contributed by Seokchan Yoon. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:38:19 +05:30
Xing Lin
80fefd093f
HDFS-17067 Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy (#5803) 2023-07-20 10:46:41 -07:00
zhangshuyan
c35f31640e
HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. (#5847) 
Contributed by Shuyan Zhang
 2023-07-20 11:39:13 +01:00
Steve Loughran
104a323f6f
HADOOP-18795. S3A DelegationToken plugin to expand return type of binding (#5821) 
Adds a class DelegationBindingInfo which contains binding info
beyond just the AWS credential list.


The binding class can be expanded when needed. Until then, all existing
implementations will work, as the new method
  DelegationBindingInfo deploy(AbstractS3ATokenIdentifier retrievedIdentifier)
falls back to the original methods.
 2023-07-20 11:27:58 +01:00
K0K0V0K
82c8070e93
YARN-9877 - Intermittent TIME_OUT of LogAggregationReport (#5784) 2023-07-20 06:30:55 +02:00
Benjamin Teke
193ff1c24e
YARN-11533. CapacityScheduler CapacityConfigType changed in legacy queue allocation mode (#5852) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-07-20 06:26:29 +02:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-20 09:21:45 +08:00
slfan1989
23ecc32d3a
YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues. (#5816) 2023-07-19 11:51:47 -07:00
slfan1989
84dd624768
YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. (#5841) 2023-07-19 11:50:38 -07:00
Moditha Hewasinghage
b6b259066f
HADOOP-18757. S3A Committer only finalizes the commits in a single thread (#5706) 
Contributed by Moditha Hewasinghage
 2023-07-19 10:03:41 +01:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) 2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) 2023-07-19 09:51:49 +08:00
Steve Loughran
b3130056f5
HADOOP-18808. LogExactlyOnce to add a debug() method (#5850) 
Contributed by Steve Loughran
 2023-07-18 14:23:19 +01:00
Viraj Jasani
38ac2f7349
HADOOP-18809. S3A prefetch read/write file operations should guard channel close (#5853) 
Contributed by Viraj Jasani
 2023-07-18 14:16:12 +01:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-16 11:27:31 +05:30
hfutatzhanghb
b95595158f
HADOOP-18801. Delete path directly when it can not be parsed in trash. (#5744). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-16 12:20:46 +08:00
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 21:13:47 +05:30
Viraj Jasani
e7d74f3d59
HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache (#5754) 
Contributed by Viraj Jasani
 2023-07-14 10:21:01 +01:00
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 00:58:43 +05:30
Xianming Lei
325f7e5fdb
YARN-11529 Add metrics for ContainerMonitorImpl. (#5828) 2023-07-13 09:50:41 +08:00
slfan1989
680af87377
YARN-11515. [Federation] Improve DefaultRequestInterceptor#init Code. (#5752) 2023-07-11 18:47:59 -07:00
slfan1989
8b88e9f8f4
YARN-11509. The FederationInterceptor#launchUAM Added retry logic. (#5727) 2023-07-11 18:47:07 -07:00
slfan1989
33b1677e9e
YARN-11524. Improve the Policy Description in Federation.md. (#5797) 2023-07-11 09:54:06 -07:00
Susheel Gupta
c13d92996d
YARN-11464: TestFSQueueConverter#testAutoCreateV2FlagsInWeightMode has a missing dot before auto-queue-creation-v2.enabled for method call assertNoValueForQueues (#5792) 
Co-authored-by: Susheel Gupta <susheel.gupta@cloudera.com>
 2023-07-11 15:20:51 +02:00
Mehakmeet Singh
fac7d26c5d
HADOOP-18781. ABFS backReference passed down to streams to avoid GC closing the FS. (#5780) 
To avoid the ABFS instance getting closed due to GC while the streams are working, attach the ABFS instance to a backReference opaque object and passing down to the streams so that we have a hard reference while the streams are working. 

Contributed by: Mehakmeet Singh
 2023-07-11 17:57:05 +05:30
WangYuanben
6843f8e4e0
HADOOP-18794. ipc.server.handler.queue.size missing from core-default.xml (#5819). Contributed by WangYuanben. 
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 16:39:50 +05:30
huhaiyang
4e699f0383
HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode (#5824). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 13:16:06 +05:30
Masatake Iwasaki
a822a3c70b
YARN-11528. Lock triple-beam to the version compatible with node.js 12 to avoid compilation error. (#5827). Contributed by Masatake Iwasaki 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 12:08:09 +05:30
WangYuanben
37b2d36edd
HDFS-17033. Update fsck to display stale state info of blocks accurately. (#5815). Contributed by Wang Yuanben. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-10 18:42:37 +08:00
huhaiyang
a84284e974
HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter (#5794). Contributed by Haiyang Hu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 13:08:47 +05:30
hfutatzhanghb
b673ebfec2
HDFS-17070. Remove unused import in DataNodeMetricHelper.java. (#5812). Contributed by farmmamba. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 12:58:23 +05:30
huhaiyang
3f983e49e0
HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" (#5817). Contributed by Haiyang Hu. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-09 23:04:20 +08:00
Harunobu Daikoku
e3683a954f
HADOOP-18793. S3A StagingCommitter does not clean up staging-uploads directory (#5818) 
Contributed by Harunobu Daikoku
 2023-07-08 12:53:54 +01:00
Tamas Domok
a36d8adfd1
YARN-11000. Replace queue resource calculation logic in updateClusterResource (#5783) 
Contributed by: Andras Gyori, Tamas Domok, Bence Kosztolnik and Benjamin Teke

---------

Co-authored-by: Benjamin Teke <benjamin.teke@gmail.com>
 2023-07-06 10:18:14 +02:00