big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,093 Commits 2 Branches 2 Tags 264 MiB
6ebce65ae8
Commit Graph

27093 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhiubok
52d66c2c11
HDFS-17136. Fix annotation description and typo in BlockPlacementPolicyDefault Class (#5907) 
Co-authored-by: huangzhaobo <huangzhaobo99@126.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:34:49 +08:00
huhaiyang
6cc56d40f5
HDFS-17135. Update fsck -blockId to display excess state info of blocks (#5904) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-31 09:21:22 +08:00
Yang Jiandan
688f76a822
YARN-11538. CS UI: queue filter do not work as expected when submitti… (#5890) 2023-07-31 07:39:42 +08:00
slfan1989
5056c267e6
YARN-11508. [Minor] Improve UnmanagedAMPoolManager/UnmanagedApplicationManager Code (#5726) 2023-07-28 20:20:23 +08:00
Peter Szucs
aae5527c9a
MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method (#5895) 2023-07-28 14:16:03 +02:00
huhaiyang
87c036efb3
HDFS-17116. RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService (#5876). Contributed by Haiyang Hu. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-28 15:15:00 +08:00
slfan1989
6d32a06125
YARN-7707. BackPort [GPG] Policy generator framework. (#5810) 2023-07-28 13:41:27 +08:00
Benjamin Teke
6dafd53626
YARN-11520. Support capacity vector for AQCv2 dynamic templates. (#5871) 2023-07-27 17:33:19 +02:00
Tamas Domok
d1c52bffbd
YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode. (#5883) 2023-07-27 15:16:34 +02:00
Sadanand Shenoy
b971222372
HDFS-17120. Support snapshot diff based copylisting for flat paths. (#5885) 2023-07-27 00:53:57 -07:00
Benjamin Teke
b1fc00d4b2
YARN-11539. Fix leaf-templates in Flexible AQC. (#5868) 2023-07-27 09:24:35 +02:00
huhaiyang
1d09dcc614
HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. (#5877). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-27 11:28:30 +08:00
ahmarsuhail
24f5f708df
HADOOP-18778. Fixes failing tests when CSE is enabled. (#5763) 
Contributed By: Ahmar Suhail <ahmarsu@amazon.co.uk>
 2023-07-26 21:56:49 +05:30
zhtttylz
068d8c7e4d
HDFS-17115. HttpFS Add Support getErasureCodeCodecs API (#5875). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-26 17:40:08 +05:30
Simbarashe Dzinamarira
ad001c93cf
HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled. (#5860). Contributed by Simbarashe Dzinamarira. 2023-07-26 13:22:13 +05:30
Ayush Saxena
c04a17f116
HADOOP-18823. Add Labeler Github Action. (#5874). Contributed by Ayush Saxena. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-25 03:04:49 +05:30
Viraj Jasani
90793e1bce
HADOOP-18805. S3A prefetch tests to work with small files (#5851) 
Contributed by Viraj Jasani
 2023-07-24 19:36:57 +01:00
yu liang
caf1816e0e
MAPREDUCE-7442. Exception message is not intusive when accessing the job configuration web UI (#5848) 2023-07-24 21:40:36 +08:00
Zhaohui Wang
9cfe9ccd26
HDFS-17119. RBF: Logger fix for StateStoreMySQLImpl. (#5882). Contributed by Zhaohui Wang. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-24 17:26:23 +05:30
zhangshuyan
49c98da838
HDFS-17112. Show decommission duration in JMX and HTML. (#5866). Contributed by Shuyan Zhang. 
Reviewed-by: hfutatzhanghb <1036798979@qq.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-24 14:38:00 +08:00
Peter Szucs
4d4b099309
YARN-11534. Fixed exception handling when container signalling is interrupted (#5864) 2023-07-21 12:30:55 +02:00
hfutatzhanghb
130bd033d5
HDFS-17074. Remove incorrect comment in TestRedudantBlocks#setup. (#5822). Contributed by farmmamba. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:45:40 +05:30
yoonsch217
fbe9a29246
YARN-11540. Fix typo: form -> from (#5861). Contributed by Seokchan Yoon. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-21 00:38:19 +05:30
Xing Lin
80fefd093f
HDFS-17067 Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy (#5803) 2023-07-20 10:46:41 -07:00
zhangshuyan
c35f31640e
HADOOP-18807. Close child file systems in ViewFileSystem when cache is disabled. (#5847) 
Contributed by Shuyan Zhang
 2023-07-20 11:39:13 +01:00
Steve Loughran
104a323f6f
HADOOP-18795. S3A DelegationToken plugin to expand return type of binding (#5821) 
Adds a class DelegationBindingInfo which contains binding info
beyond just the AWS credential list.


The binding class can be expanded when needed. Until then, all existing
implementations will work, as the new method
  DelegationBindingInfo deploy(AbstractS3ATokenIdentifier retrievedIdentifier)
falls back to the original methods.
 2023-07-20 11:27:58 +01:00
K0K0V0K
82c8070e93
YARN-9877 - Intermittent TIME_OUT of LogAggregationReport (#5784) 2023-07-20 06:30:55 +02:00
Benjamin Teke
193ff1c24e
YARN-11533. CapacityScheduler CapacityConfigType changed in legacy queue allocation mode (#5852) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-07-20 06:26:29 +02:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-20 09:21:45 +08:00
slfan1989
23ecc32d3a
YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues. (#5816) 2023-07-19 11:51:47 -07:00
slfan1989
84dd624768
YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. (#5841) 2023-07-19 11:50:38 -07:00
Moditha Hewasinghage
b6b259066f
HADOOP-18757. S3A Committer only finalizes the commits in a single thread (#5706) 
Contributed by Moditha Hewasinghage
 2023-07-19 10:03:41 +01:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) 2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) 2023-07-19 09:51:49 +08:00
Steve Loughran
b3130056f5
HADOOP-18808. LogExactlyOnce to add a debug() method (#5850) 
Contributed by Steve Loughran
 2023-07-18 14:23:19 +01:00
Viraj Jasani
38ac2f7349
HADOOP-18809. S3A prefetch read/write file operations should guard channel close (#5853) 
Contributed by Viraj Jasani
 2023-07-18 14:16:12 +01:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-16 11:27:31 +05:30
hfutatzhanghb
b95595158f
HADOOP-18801. Delete path directly when it can not be parsed in trash. (#5744). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-16 12:20:46 +08:00
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 21:13:47 +05:30
Viraj Jasani
e7d74f3d59
HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache (#5754) 
Contributed by Viraj Jasani
 2023-07-14 10:21:01 +01:00
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 00:58:43 +05:30
Xianming Lei
325f7e5fdb
YARN-11529 Add metrics for ContainerMonitorImpl. (#5828) 2023-07-13 09:50:41 +08:00
slfan1989
680af87377
YARN-11515. [Federation] Improve DefaultRequestInterceptor#init Code. (#5752) 2023-07-11 18:47:59 -07:00
slfan1989
8b88e9f8f4
YARN-11509. The FederationInterceptor#launchUAM Added retry logic. (#5727) 2023-07-11 18:47:07 -07:00
slfan1989
33b1677e9e
YARN-11524. Improve the Policy Description in Federation.md. (#5797) 2023-07-11 09:54:06 -07:00
Susheel Gupta
c13d92996d
YARN-11464: TestFSQueueConverter#testAutoCreateV2FlagsInWeightMode has a missing dot before auto-queue-creation-v2.enabled for method call assertNoValueForQueues (#5792) 
Co-authored-by: Susheel Gupta <susheel.gupta@cloudera.com>
 2023-07-11 15:20:51 +02:00
Mehakmeet Singh
fac7d26c5d
HADOOP-18781. ABFS backReference passed down to streams to avoid GC closing the FS. (#5780) 
To avoid the ABFS instance getting closed due to GC while the streams are working, attach the ABFS instance to a backReference opaque object and passing down to the streams so that we have a hard reference while the streams are working. 

Contributed by: Mehakmeet Singh
 2023-07-11 17:57:05 +05:30