big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,766 Commits 2 Branches 2 Tags 264 MiB
193ff1c24e
Commit Graph

26766 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Benjamin Teke
193ff1c24e
YARN-11533. CapacityScheduler CapacityConfigType changed in legacy queue allocation mode (#5852) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-07-20 06:26:29 +02:00
zhangshuyan
7ba2bd6305
HDFS-17094. EC: Fix bug in block recovery when there are stale datanodes. (#5854) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-07-20 09:21:45 +08:00
slfan1989
23ecc32d3a
YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues. (#5816) 2023-07-19 11:51:47 -07:00
slfan1989
84dd624768
YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. (#5841) 2023-07-19 11:50:38 -07:00
Moditha Hewasinghage
b6b259066f
HADOOP-18757. S3A Committer only finalizes the commits in a single thread (#5706) 
Contributed by Moditha Hewasinghage
 2023-07-19 10:03:41 +01:00
huhaiyang
8df1616f53
HDFS-17088. Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md (#5846) 2023-07-19 09:53:51 +08:00
WangYuanben
0fe1863094
HDFS-17091. Blocks on DECOMMISSIONING DNs should be sorted properly in LocatedBlocks. (#5849) 2023-07-19 09:51:49 +08:00
Steve Loughran
b3130056f5
HADOOP-18808. LogExactlyOnce to add a debug() method (#5850) 
Contributed by Steve Loughran
 2023-07-18 14:23:19 +01:00
Viraj Jasani
38ac2f7349
HADOOP-18809. S3A prefetch read/write file operations should guard channel close (#5853) 
Contributed by Viraj Jasani
 2023-07-18 14:16:12 +01:00
huhaiyang
c44823dadb
HDFS-17075. Reconfig disk balancer parameters for datanode (#5823). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-16 11:27:31 +05:30
hfutatzhanghb
b95595158f
HADOOP-18801. Delete path directly when it can not be parsed in trash. (#5744). Contributed by farmmamba. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-16 12:20:46 +08:00
huhaiyang
8dd9c874e1
HDFS-17086. Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota (#5842). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-15 12:00:07 +05:30
hfutatzhanghb
c457c445b7
HDFS-17068. Datanode should record last directory scan time. (#5809). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 21:13:47 +05:30
Viraj Jasani
e7d74f3d59
HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache (#5754) 
Contributed by Viraj Jasani
 2023-07-14 10:21:01 +01:00
huhaiyang
c1d19c3951
HDFS-17081. EC: Add logic for striped blocks in isSufficientlyReplicated (#5833). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:56:54 +05:30
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
huhaiyang
4c8d048f25
HDFS-17082. Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md" (#5834). Contributed by Haiyang Hu. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 00:58:43 +05:30
Xianming Lei
325f7e5fdb
YARN-11529 Add metrics for ContainerMonitorImpl. (#5828) 2023-07-13 09:50:41 +08:00
slfan1989
680af87377
YARN-11515. [Federation] Improve DefaultRequestInterceptor#init Code. (#5752) 2023-07-11 18:47:59 -07:00
slfan1989
8b88e9f8f4
YARN-11509. The FederationInterceptor#launchUAM Added retry logic. (#5727) 2023-07-11 18:47:07 -07:00
slfan1989
33b1677e9e
YARN-11524. Improve the Policy Description in Federation.md. (#5797) 2023-07-11 09:54:06 -07:00
Susheel Gupta
c13d92996d
YARN-11464: TestFSQueueConverter#testAutoCreateV2FlagsInWeightMode has a missing dot before auto-queue-creation-v2.enabled for method call assertNoValueForQueues (#5792) 
Co-authored-by: Susheel Gupta <susheel.gupta@cloudera.com>
 2023-07-11 15:20:51 +02:00
Mehakmeet Singh
fac7d26c5d
HADOOP-18781. ABFS backReference passed down to streams to avoid GC closing the FS. (#5780) 
To avoid the ABFS instance getting closed due to GC while the streams are working, attach the ABFS instance to a backReference opaque object and passing down to the streams so that we have a hard reference while the streams are working. 

Contributed by: Mehakmeet Singh
 2023-07-11 17:57:05 +05:30
WangYuanben
6843f8e4e0
HADOOP-18794. ipc.server.handler.queue.size missing from core-default.xml (#5819). Contributed by WangYuanben. 
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 16:39:50 +05:30
huhaiyang
4e699f0383
HDFS-17076. Remove the unused method isSlownodeByNameserviceId in DataNode (#5824). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 13:16:06 +05:30
Masatake Iwasaki
a822a3c70b
YARN-11528. Lock triple-beam to the version compatible with node.js 12 to avoid compilation error. (#5827). Contributed by Masatake Iwasaki 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-11 12:08:09 +05:30
WangYuanben
37b2d36edd
HDFS-17033. Update fsck to display stale state info of blocks accurately. (#5815). Contributed by Wang Yuanben. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Reviewed-by: Hualong Zhang <hualong.z@hotmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-10 18:42:37 +08:00
huhaiyang
a84284e974
HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter (#5794). Contributed by Haiyang Hu. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 13:08:47 +05:30
hfutatzhanghb
b673ebfec2
HDFS-17070. Remove unused import in DataNodeMetricHelper.java. (#5812). Contributed by farmmamba. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-10 12:58:23 +05:30
huhaiyang
3f983e49e0
HDFS-17073. Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute" (#5817). Contributed by Haiyang Hu. 
Reviewed-by: Xing Lin <linxingnku@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-09 23:04:20 +08:00
Harunobu Daikoku
e3683a954f
HADOOP-18793. S3A StagingCommitter does not clean up staging-uploads directory (#5818) 
Contributed by Harunobu Daikoku
 2023-07-08 12:53:54 +01:00
Tamas Domok
a36d8adfd1
YARN-11000. Replace queue resource calculation logic in updateClusterResource (#5783) 
Contributed by: Andras Gyori, Tamas Domok, Bence Kosztolnik and Benjamin Teke

---------

Co-authored-by: Benjamin Teke <benjamin.teke@gmail.com>
 2023-07-06 10:18:14 +02:00
yl09099
178d046ce6
YARN-11526. [Client] Add a unit test to FormattingCLIUtils (#5801) 2023-07-06 09:11:13 +08:00
slfan1989
6042d59904
YARN-11090. [GPG] Support Secure Mode. (#5782) 2023-07-03 08:56:24 -07:00
slfan1989
e8590adb7b
HADOOP-18779. Improve hadoop-function.sh#status script. (#5762) 2023-07-03 08:46:57 -07:00
Zhaohui Wang
9659d0720d
HDFS-17065. Fix typos in hadoop-hdfs-project (#5796) 2023-07-03 08:41:37 -07:00
PJ Fanning
5a35fb5a72
HADOOP-18783. Upgrade to netty 4.1.94 due to CVE (#5774). Contributed by PJ Fanning. 2023-07-02 14:08:13 +05:30
zhtttylz
750c0fc631
HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack. (#5759). Contributed by Hualong Zhang and Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-07-02 15:37:19 +08:00
Hexiaoqiao
59a7836d13
HADOOP-18789. Remove ozone from hadoop dev support. (#5800). Contributed by Xiaoqiao He. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2023-07-02 15:23:32 +08:00
wangzhaohui
8fb611c1f7
HDFS-17062. HA NameNode Web UI should show last HA Transition time (#5791). Contributed by Zhaohui Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-30 08:56:34 +05:30
huhaiyang
4bd873b816
HDFS-17044. Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode. (#5735). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-28 12:03:15 +08:00
slfan1989
8a52990150
YARN-11519. [Federation] Add RouterAuditLog to log4j.properties. (#5785) 2023-06-27 10:52:59 -07:00
Mehakmeet Singh
5db7107b77
HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications (#5737) 
Changing fs.azure.buffer.dir for azure so things clean up better in long-lived yarn clusters.

Contributed by: Mehakmeet Singh
 2023-06-27 20:22:00 +05:30
Steve Loughran
0d057e27c3
MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378) 
By default, the mapreduce manifest committer is used for jobs working with abfs and gcs.
Hadoop mapreduce will pick this up automatically; for Spark it is a bit complicated: read the docs
to see the steps required.
 2023-06-27 13:55:20 +01:00
PJ Fanning
56ef05a9ca
HADOOP-18782. Upgrade to snappy-java 1.1.10.1 due to CVEs (#5773) 
Addresses CVE-2023-34454

Contributed by PJ Fanning
 2023-06-27 11:53:02 +01:00
rohit-kb
21d9c4727c
HADOOP-18773. Upgrade maven-shade-plugin to 3.4.1 (#5750) 
This is needed to successfully shade the hadoop binaries on recent Java versions.

Contributed by Rohit Kumar
 2023-06-27 10:40:22 +01:00
Xing Lin
03902f5ef0
HDFS-17055 Export HAState as a metric from Namenode for monitoring (#5764) 2023-06-26 15:52:59 -07:00
Gautham B A
a85272c33d
HADOOP-18751. Fix incorrect output path in javadoc build phase (#5688) 2023-06-26 15:52:17 -07:00
slfan1989
a4cf4c3778
YARN-11517. Improve Federation#RouterCLI DeregisterSubCluster Code. (#5766) 2023-06-26 09:43:49 -07:00
huhaiyang
161f80810f
HDFS-17057. Add DataNode maintenance states to Federation UI (#5775) 2023-06-26 09:38:57 -07:00