big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,453 Commits 2 Branches 2 Tags 264 MiB
f931ede86b
Commit Graph

27453 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Takanobu Asanuma
c4f7a3625b
HDFS-17435. Fix TestRouterRpc failed (#6650) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-03-26 09:38:35 +09:00
Gautham B A
76489e579b
HADOOP-19127. Do not run unit tests on Windows pre-commit CI (#6672) 2024-03-25 09:16:03 -07:00
Gautham B A
44a249e32a
Exclude files from Apache RAT (#6671) 2024-03-25 09:13:57 -07:00
PJ Fanning
7653f968e5
HADOOP-19116. Update to zookeeper client 3.8.4 due to CVE-2024-23944. (#6638) 
Updated ZK client dependency to 3.8.4 to address  CVE-2024-23944.

Contributed by PJ Fanning
 2024-03-25 15:10:56 +00:00
Anuj Modi
c4fa1b65fb
HADOOP-19089: [ABFS] Reverting Back Support of setXAttr() and getXAttr() on root path (#6592) 
This reverts most of
HADOOP-18869: [ABFS] Fix behavior of a File System APIs on root path (#6003).

Calling getXAttr("/") or setXAttr("/") on an abfs container will fail with

`Operation failed: "The request URI is invalid.", HTTP 400 Bad Request`

 
This change is to ensure:
* Consistency across ADLS clients
* Consistency across authentication mechanisms.

Contributed by Anuj Modi
 2024-03-25 14:13:24 +00:00
Alex
5c7e40f910
HADOOP-19111. Removing redundant debug message about client info (#6666). Contributed by Zhongkun Wu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-25 11:44:33 +08:00
yu liang
55dca911cc
HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time (#6587) Contributed by liangyu. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-24 10:14:27 +08:00
LiuGuH
a60b5e2de3
MAPREDUCE-7469. NNBench createControlFiles should use thread pool to improve performance. (#6463) Contributed by liuguanghua. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-23 06:15:56 +08:00
huhaiyang
8cd4704e0a
HDFS-17430. RecoveringBlock will skip no live replicas when get block recovery command. (#6635) 2024-03-22 09:43:12 -04:00
XiaobaoWu
062836c020
HDFS-17436. Supplement log information for AccessControlException (#6651) 2024-03-22 11:21:03 +08:00
XiaobaoWu
a375ef8cfa
YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata (#6616) 
Co-authored-by: wuxiaobao <xbaowu@163.com>
 2024-03-21 03:12:14 -04:00
Takanobu Asanuma
adab3a22aa
HDFS-17432. Fix junit dependency to enable JUnit4 tests to run in hadoop-hdfs-rbf (#6639) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2024-03-21 14:53:02 +09:00
hfutatzhanghb
2eb7246ea7
HDFS-17433. metrics sumOfActorCommandQueueLength should only record valid commands. (#6644) 2024-03-20 23:41:35 -04:00
huhaiyang
77c600d769
HDFS-17426. Remove Invalid FileSystemECReadStats logic in DFSInputStream (#6628) 2024-03-21 10:33:41 +08:00
Peter Szucs
a957cd5049
YARN-5305. Allow log aggregation to discard expired delegation tokens (#6625) 2024-03-20 15:33:10 +01:00
huhaiyang
12a26d8b19
HDFS-17431. Fix log format for BlockRecoveryWorker#recoverBlocks (#6643) 2024-03-19 23:22:45 -04:00
Adnan Hemani
8b2058a4e7
HADOOP-19050. S3A: Support S3 Access Grants (#6544) 
This adds support for Amazon S3 Access Grants to the S3A connector.

For more information, see:
* https://aws.amazon.com/s3/features/access-grants/
* https://github.com/aws/aws-s3-accessgrants-plugin-java-v2/

Contributed by Adnan Hemani
 2024-03-19 17:49:51 +00:00
Steve Loughran
705fb8323b
HADOOP-19119. Spotbugs: possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize() (#6642) 
Spotbugs is mistaken here as it doesn't observer the read/write locks used
to manage exclusive access to the maps.

* cache the value between checks
* tag as @VisibleForTesting

Contributed by Steve Loughran
 2024-03-19 17:18:07 +00:00
slfan1989
ff3f2255d2
HADOOP-19112. Hadoop 3.4.0 release wrap-up. (#6640) Contributed by Shilun Fan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-19 20:08:03 +08:00
Vinayakumar B
0f51d2a4ec
HADOOP-14451. Deadlock in NativeIO (#6632) 2024-03-18 10:53:21 +05:30
Tsz-Wo Nicholas Sze
b25b28e5bb
HDFS-17380. FsImageValidation: remove inaccessible nodes. (#6549). Contributed by Tsz-wo Sze. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-18 11:52:17 +08:00
drankye
4d88f9892a HADOOP-19085. Compatibility Benchmark over HCFS Implementations 
Contributed by Han Liu
 2024-03-17 16:48:29 +08:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-15 08:14:38 +08:00
Viraj Jasani
a325876fec
HADOOP-19066. Run FIPS test for valid bucket locations (ADDENDUM) (#6624) 
FIPS is only supported in north america AWS regions; relevant tests in
ITestS3AEndpointRegion are skipped for buckets with different endpoints/regions.
 2024-03-13 13:21:50 +00:00
PJ Fanning
e28c78f9a2
HADOOP-19088. Use jersey-json 1.22.0 (#6585) 
Contributed by pjfanning
 2024-03-12 20:16:47 +00:00
Viraj Jasani
44c14edac7
HADOOP-19066. S3A: AWS SDK V2 - Enabling FIPS should be allowed with central endpoint (#6539) 
Contributed by Viraj Jasani
 2024-03-12 18:49:06 +00:00
Lei313
e211f6f83d
HDFS-17391. Adjust the checkpoint io buffer size to the chunk size (#6594). Contributed by lei w. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-12 18:36:43 +08:00
Lei313
dbf08c872a
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-03-12 12:25:33 +08:00
Steve Loughran
bb32aec88e
HADOOP-19043. S3A: Regression: ITestS3AOpenCost fails on prefetch test runs (#6465) 
Disables the new tests added in:

HADOOP-19027. S3A: S3AInputStream doesn't recover from HTTP/channel exceptions #6425

The underlying issue here is that the block prefetch code can identify
when there's a mismatch between declared and actual length, and doesn't
store any of the incomplete buffer.

This should be addressed in HADOOP-18184.

Contributed by Steve Loughran
 2024-03-08 12:48:38 +00:00
PJ Fanning
fc166d3aec
HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fanning. 2024-03-07 15:09:01 +05:30
ritegarg
58afe43769
HDFS-17299. Adding rack failure tolerance when creating a new file (#6566) 2024-03-06 13:08:05 -08:00
hfutatzhanghb
7012986fc3
HDFS-17345. Add a metrics to record block report generating cost time. (#6475). Contributed by farmmamba. 
Reviewed-by:  Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-03-06 16:59:00 +08:00
dannytbecker
5584efd8d4
HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade (#6583) 2024-03-05 11:31:31 -08:00
Jtdellaringa
a2d7241190
HDFS-17404. Add NN Socket Address to log when processing command from active NN (#6598) 
* Add nnSocketAddress to log when processing command from active nn
 2024-03-05 10:33:22 -08:00
Steve Loughran
dae871e3e0
YARN-11657. Remove protobuf-2.5 from hadoop-yarn-api module (#6575) (#6580) 
The import of protobuf-java-2.5 in the hadoop-yarn-api module
is downgraded from "compile" to "provided"

This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar

It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar

Contributed by Steve Loughran
 2024-03-05 11:01:14 +00:00
HarshitGupta11
d974a12f39
HADOOP-19082: S3A: Update AWS SDK V2 to 2.24.6 (#6568) 
Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module.

Contributed by Harshit Gupta
 2024-03-05 10:15:05 +00:00
Steve Loughran
095229fefb
HADOOP-19097. S3A: Set fs.s3a.connection.establish.timeout to 30s (#6601) 
This is consistent with the value in the hadoop-aws source code

Contributed by Steve Loughran
 2024-03-05 10:10:27 +00:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. (#6430) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-03-02 21:35:24 +09:00
Steve Loughran
a0ce2170db
HADOOP-19084. Prune hadoop-common transitive dependencies (#6574) (#6582) 
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
    
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging

This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.

Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.

Contributed by Steve Loughran
 2024-03-01 12:51:13 +00:00
huhaiyang
34bb28649c
HDFS-17406. Suppress UnresolvedPathException in hdfs router log (#6603) 2024-03-01 19:03:41 +08:00
hfutatzhanghb
15af52954f
HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. (#6509). Contributed by farmmamba. 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-02-27 10:19:57 +08:00
ZanderXu
a897e745f5
HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem (#6567) 2024-02-23 15:37:27 +08:00
slfan1989
10ab8abccd
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)" (#6578) 
This reverts commit 555faf28ce.
 2024-02-23 14:25:15 +08:00
Anuj Modi
99b9e7fb43
HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport (#6069) 
Contributed By: Anuj Modi
 2024-02-22 11:49:37 -06:00
huangzhaobo
6dfebdbce4
HADOOP-19065. Update Protocol Buffers installation to 3.21.12 (#6526) 
Instructions and docker script for installing protobuf updated from 3.7.1 to 3.21.12

Contributed by huangzhaobo
 2024-02-22 17:09:46 +00:00
Lars Francke
d1daf26b85
Fixes HDFS-17181 by routing all CREATE requests to the BlockManager (#6108) 2024-02-20 10:58:49 -08:00
Anuj Modi
1336c362e5
Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. (#5881) 
Contributed By: Anuj Modi
 2024-02-20 11:31:42 -06:00
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes (#6508) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-18 09:15:40 +08:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up (#6498) 2024-02-13 16:24:23 -08:00