Commit Graph

27292 Commits

Author SHA1 Message Date
Takanobu Asanuma
adab3a22aa
HDFS-17432. Fix junit dependency to enable JUnit4 tests to run in hadoop-hdfs-rbf (#6639)
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2024-03-21 14:53:02 +09:00
hfutatzhanghb
2eb7246ea7
HDFS-17433. metrics sumOfActorCommandQueueLength should only record valid commands. (#6644) 2024-03-20 23:41:35 -04:00
huhaiyang
77c600d769
HDFS-17426. Remove Invalid FileSystemECReadStats logic in DFSInputStream (#6628) 2024-03-21 10:33:41 +08:00
Peter Szucs
a957cd5049
YARN-5305. Allow log aggregation to discard expired delegation tokens (#6625) 2024-03-20 15:33:10 +01:00
huhaiyang
12a26d8b19
HDFS-17431. Fix log format for BlockRecoveryWorker#recoverBlocks (#6643) 2024-03-19 23:22:45 -04:00
Adnan Hemani
8b2058a4e7
HADOOP-19050. S3A: Support S3 Access Grants (#6544)
This adds support for Amazon S3 Access Grants to the S3A connector.

For more information, see:
* https://aws.amazon.com/s3/features/access-grants/
* https://github.com/aws/aws-s3-accessgrants-plugin-java-v2/

Contributed by Adnan Hemani
2024-03-19 17:49:51 +00:00
Steve Loughran
705fb8323b
HADOOP-19119. Spotbugs: possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize() (#6642)
Spotbugs is mistaken here as it doesn't observer the read/write locks used
to manage exclusive access to the maps.

* cache the value between checks
* tag as @VisibleForTesting

Contributed by Steve Loughran
2024-03-19 17:18:07 +00:00
slfan1989
ff3f2255d2
HADOOP-19112. Hadoop 3.4.0 release wrap-up. (#6640) Contributed by Shilun Fan.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-19 20:08:03 +08:00
Vinayakumar B
0f51d2a4ec
HADOOP-14451. Deadlock in NativeIO (#6632) 2024-03-18 10:53:21 +05:30
Tsz-Wo Nicholas Sze
b25b28e5bb
HDFS-17380. FsImageValidation: remove inaccessible nodes. (#6549). Contributed by Tsz-wo Sze.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-18 11:52:17 +08:00
drankye
4d88f9892a HADOOP-19085. Compatibility Benchmark over HCFS Implementations
Contributed by Han Liu
2024-03-17 16:48:29 +08:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang.
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-15 08:14:38 +08:00
Viraj Jasani
a325876fec
HADOOP-19066. Run FIPS test for valid bucket locations (ADDENDUM) (#6624)
FIPS is only supported in north america AWS regions; relevant tests in
ITestS3AEndpointRegion are skipped for buckets with different endpoints/regions.
2024-03-13 13:21:50 +00:00
PJ Fanning
e28c78f9a2
HADOOP-19088. Use jersey-json 1.22.0 (#6585)
Contributed by pjfanning
2024-03-12 20:16:47 +00:00
Viraj Jasani
44c14edac7
HADOOP-19066. S3A: AWS SDK V2 - Enabling FIPS should be allowed with central endpoint (#6539)
Contributed by Viraj Jasani
2024-03-12 18:49:06 +00:00
Lei313
e211f6f83d
HDFS-17391. Adjust the checkpoint io buffer size to the chunk size (#6594). Contributed by lei w.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-12 18:36:43 +08:00
Lei313
dbf08c872a
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-12 12:25:33 +08:00
Steve Loughran
bb32aec88e
HADOOP-19043. S3A: Regression: ITestS3AOpenCost fails on prefetch test runs (#6465)
Disables the new tests added in:

HADOOP-19027. S3A: S3AInputStream doesn't recover from HTTP/channel exceptions #6425

The underlying issue here is that the block prefetch code can identify
when there's a mismatch between declared and actual length, and doesn't
store any of the incomplete buffer.

This should be addressed in HADOOP-18184.

Contributed by Steve Loughran
2024-03-08 12:48:38 +00:00
PJ Fanning
fc166d3aec
HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fanning. 2024-03-07 15:09:01 +05:30
ritegarg
58afe43769
HDFS-17299. Adding rack failure tolerance when creating a new file (#6566) 2024-03-06 13:08:05 -08:00
hfutatzhanghb
7012986fc3
HDFS-17345. Add a metrics to record block report generating cost time. (#6475). Contributed by farmmamba.
Reviewed-by:  Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-03-06 16:59:00 +08:00
dannytbecker
5584efd8d4
HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade (#6583) 2024-03-05 11:31:31 -08:00
Jtdellaringa
a2d7241190
HDFS-17404. Add NN Socket Address to log when processing command from active NN (#6598)
* Add nnSocketAddress to log when processing command from active nn
2024-03-05 10:33:22 -08:00
Steve Loughran
dae871e3e0
YARN-11657. Remove protobuf-2.5 from hadoop-yarn-api module (#6575) (#6580)
The import of protobuf-java-2.5 in the hadoop-yarn-api module
is downgraded from "compile" to "provided"

This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar

It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar

Contributed by Steve Loughran
2024-03-05 11:01:14 +00:00
HarshitGupta11
d974a12f39
HADOOP-19082: S3A: Update AWS SDK V2 to 2.24.6 (#6568)
Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module.

Contributed by Harshit Gupta
2024-03-05 10:15:05 +00:00
Steve Loughran
095229fefb
HADOOP-19097. S3A: Set fs.s3a.connection.establish.timeout to 30s (#6601)
This is consistent with the value in the hadoop-aws source code

Contributed by Steve Loughran
2024-03-05 10:10:27 +00:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. (#6430)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-03-02 21:35:24 +09:00
Steve Loughran
a0ce2170db
HADOOP-19084. Prune hadoop-common transitive dependencies (#6574) (#6582)
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
    
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging

This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.

Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.

Contributed by Steve Loughran
2024-03-01 12:51:13 +00:00
huhaiyang
34bb28649c
HDFS-17406. Suppress UnresolvedPathException in hdfs router log (#6603) 2024-03-01 19:03:41 +08:00
hfutatzhanghb
15af52954f
HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. (#6509). Contributed by farmmamba.
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-02-27 10:19:57 +08:00
ZanderXu
a897e745f5
HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem (#6567) 2024-02-23 15:37:27 +08:00
slfan1989
10ab8abccd
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537)" (#6578)
This reverts commit 555faf28ce.
2024-02-23 14:25:15 +08:00
Anuj Modi
99b9e7fb43
HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport (#6069)
Contributed By: Anuj Modi
2024-02-22 11:49:37 -06:00
huangzhaobo
6dfebdbce4
HADOOP-19065. Update Protocol Buffers installation to 3.21.12 (#6526)
Instructions and docker script for installing protobuf updated from 3.7.1 to 3.21.12

Contributed by huangzhaobo
2024-02-22 17:09:46 +00:00
Lars Francke
d1daf26b85
Fixes HDFS-17181 by routing all CREATE requests to the BlockManager (#6108) 2024-02-20 10:58:49 -08:00
Anuj Modi
1336c362e5
Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. (#5881)
Contributed By: Anuj Modi
2024-02-20 11:31:42 -06:00
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes (#6508)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. (#6504) Contributed by Hualong Zhang.
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-18 09:15:40 +08:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up (#6498) 2024-02-13 16:24:23 -08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052)
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
2024-02-13 16:33:51 +00:00
Steve Loughran
7651afd3db
HADOOP-19057. S3A: Landsat bucket used in tests no longer accessible (#6515)
The AWS landsat data previously used in some S3A tests is no
longer accessible

This PR moves to the new external file
s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz

* Large enough file for scale tests
* Bucket supports anonymous access
* Ends in .gz to keep codec tests happy
* No spaces in path to keep bucket-info happy

Test Code Changes
* Leaves the test key name alone: fs.s3a.scale.test.csvfile
* Rename all methods and fields move remove "csv" from their names and
  move to "external file" we no longer require it to be CSV.
* Path definition and helper methods have been moved to PublicDatasetTestUtils
* Improve error reporting in ITestS3AInputStreamPerformance if the file
  is too short
  
With S3 Select removed, there is no need for the file to be
a CSV file; there is a test which tries to unzip it; other
tests have a minimum file size.

Consult the JIRA for the settings to add to auth-keys.xml
to switch earlier builds to this same file.

Contributed by Steve Loughran
2024-02-13 10:46:36 +00:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider (#6510)
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
2024-02-13 10:49:39 +09:00
slfan1989
555faf28ce
HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. (#6537) Contributed by Shilun Fan
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-11 07:41:46 +08:00
Sadanand Shenoy
0bf439c0f9
HDFS-17376. Distcp creates Factor 1 replication file on target if Source is EC. (#6540) 2024-02-09 17:26:32 +00:00
Adnan Hemani
50d256ef3c
HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 (#6538)
Contributed by Adnan Hemani
2024-02-08 20:38:37 +00:00
slfan1989
8011b21c52
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-08 19:18:04 +08:00
Steve Loughran
3f98cb6741
HADOOP-19045. CreateSession Timeout - followup (#6532)
This is a followup to PR:
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)

Remove all declarations of fs.s3a.connection.request.timeout
in
- hadoop-common/src/main/resources/core-default.xml
- hadoop-aws/src/test/resources/core-site.xml

New test in TestAwsClientConfig to verify that the value
defined in fs.s3a.Constants class is used.

This is brittle to someone overriding it in their test setups,
but as this test is intended to verify that the option is not
explicitly set, there's no workaround.

Contributed by Steve Loughran
2024-02-07 12:07:54 +00:00
huhaiyang
5ad7737132
HDFS-17342. Fix DataNode may invalidates normal block causing missing block (#6464). Contributed by Haiyang Hu.
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Chengwei Wang <1139557635@qq.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-02-06 17:52:52 +08:00
Takanobu Asanuma
9a7eeadaac
HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf (#6522) 2024-02-06 11:54:14 +09:00
Antonio Murgia
b11159d799
HADOOP-18993. Add option fs.s3a.classloader.isolation (#6301)
The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation;

This can assist in using custom credential providers and other extension points.

Contributed by Antonio Murgia
2024-02-05 17:59:36 +00:00