big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467)

Browse Source 
This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down
to calls to CreateSession made in the AWS SDK to get S3 Express session tokens.

Contributed by Steve Loughran
This commit is contained in:
Steve Loughran 2024-01-21 19:00:34 +00:00 committed by GitHub
parent 76691dfa14
commit d274f778c1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 15 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
LICENSE-binary
View File

@ -215,7 +215,7 @@ com.aliyun:aliyun-java-sdk-kms:2.11.0
com.aliyun:aliyun-java-sdk-ram:3.1.0 com.aliyun:aliyun-java-sdk-ram:3.1.0
com.aliyun:aliyun-java-sdk-sts:3.0.0 com.aliyun:aliyun-java-sdk-sts:3.0.0
com.aliyun.oss:aliyun-sdk-oss:3.13.2 com.aliyun.oss:aliyun-sdk-oss:3.13.2
com.amazonaws:aws-java-sdk-bundle:1.12.565 com.amazonaws:aws-java-sdk-bundle:1.12.599
com.cedarsoftware:java-util:1.9.0 com.cedarsoftware:java-util:1.9.0
com.cedarsoftware:json-io:2.5.1 com.cedarsoftware:json-io:2.5.1
com.fasterxml.jackson.core:jackson-annotations:2.12.7 com.fasterxml.jackson.core:jackson-annotations:2.12.7
@ -363,7 +363,7 @@ org.objenesis:objenesis:2.6
org.xerial.snappy:snappy-java:1.1.10.4 org.xerial.snappy:snappy-java:1.1.10.4
org.yaml:snakeyaml:2.0 org.yaml:snakeyaml:2.0
org.wildfly.openssl:wildfly-openssl:1.1.3.Final org.wildfly.openssl:wildfly-openssl:1.1.3.Final
software.amazon.awssdk:bundle:jar:2.21.41 software.amazon.awssdk:bundle:jar:2.23.5
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------

2
NOTICE-binary
View File

@ -66,7 +66,7 @@ available from http://www.digip.org/jansson/.
AWS SDK for Java AWS SDK for Java
Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved. Copyright 2010-2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
This product includes software developed by This product includes software developed by
Amazon Technologies, Inc (http://www.amazon.com/). Amazon Technologies, Inc (http://www.amazon.com/).

4
hadoop-project/pom.xml
View File

@ -186,8 +186,8 @@
<exec-maven-plugin.version>1.3.1</exec-maven-plugin.version> <exec-maven-plugin.version>1.3.1</exec-maven-plugin.version>
<make-maven-plugin.version>1.0-beta-1</make-maven-plugin.version> <make-maven-plugin.version>1.0-beta-1</make-maven-plugin.version>
<surefire.fork.timeout>900</surefire.fork.timeout> <surefire.fork.timeout>900</surefire.fork.timeout>
<aws-java-sdk.version>1.12.565</aws-java-sdk.version> <aws-java-sdk.version>1.12.599</aws-java-sdk.version>
<aws-java-sdk-v2.version>2.21.41</aws-java-sdk-v2.version> <aws-java-sdk-v2.version>2.23.5</aws-java-sdk-v2.version>
<aws.eventstream.version>1.0.1</aws.eventstream.version> <aws.eventstream.version>1.0.1</aws.eventstream.version>
<hsqldb.version>2.7.1</hsqldb.version> <hsqldb.version>2.7.1</hsqldb.version>
<frontend-maven-plugin.version>1.11.2</frontend-maven-plugin.version> <frontend-maven-plugin.version>1.11.2</frontend-maven-plugin.version>

13
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
View File

@ -1142,7 +1142,7 @@ as it may take a couple of SDK updates before it is ready.
1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/). 1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/).
1. Create a private git branch of trunk for JIRA, and in 1. Create a private git branch of trunk for JIRA, and in
`hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version. `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version.
1. Update AWS SDK versions in NOTICE.txt. 1. Update AWS SDK versions in NOTICE.txt and LICENSE.binary
1. Do a clean build and rerun all the `hadoop-aws` tests. 1. Do a clean build and rerun all the `hadoop-aws` tests.
This includes the `-Pscale` set, with a role defined for the assumed role tests. This includes the `-Pscale` set, with a role defined for the assumed role tests.
in `fs.s3a.assumed.role.arn` for testing assumed roles, in `fs.s3a.assumed.role.arn` for testing assumed roles,
@ -1164,11 +1164,18 @@ your IDE or via maven.
`mvn dependency:tree -Dverbose > target/dependencies.txt`. `mvn dependency:tree -Dverbose > target/dependencies.txt`.
Examine the `target/dependencies.txt` file to verify that no new Examine the `target/dependencies.txt` file to verify that no new
artifacts have unintentionally been declared as dependencies artifacts have unintentionally been declared as dependencies
of the shaded `aws-java-sdk-bundle` artifact. of the shaded `software.amazon.awssdk:bundle:jar` artifact.
1. Run a full AWS-test suite with S3 client-side encryption enabled by 1. Run a full AWS-test suite with S3 client-side encryption enabled by
setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS setting `fs.s3a.encryption.algorithm` to 'CSE-KMS' and setting up AWS-KMS
Key ID in `fs.s3a.encryption.key`. Key ID in `fs.s3a.encryption.key`.
The dependency chain of the `hadoop-aws` module should be similar to this, albeit
with different version numbers:
```
[INFO] +- org.apache.hadoop:hadoop-aws:jar:3.4.0-SNAPSHOT:compile
[INFO] | +- software.amazon.awssdk:bundle:jar:2.23.5:compile
[INFO] | \- org.wildfly.openssl:wildfly-openssl:jar:1.1.3.Final:compile
```
### Basic command line regression testing ### Basic command line regression testing
We need a run through of the CLI to see if there have been changes there We need a run through of the CLI to see if there have been changes there
@ -1365,5 +1372,5 @@ Don't be surprised if this happens, don't worry too much, and,
while that rollback option is there to be used, ideally try to work forwards. while that rollback option is there to be used, ideally try to work forwards.
If the problem is with the SDK, file issues with the If the problem is with the SDK, file issues with the
[AWS SDK Bug tracker](https://github.com/aws/aws-sdk-java/issues). [AWS V2 SDK Bug tracker](https://github.com/aws/aws-sdk-java-v2/issues).
If the problem can be fixed or worked around in the Hadoop code, do it there too. If the problem can be fixed or worked around in the Hadoop code, do it there too.