hadoop/hadoop-tools/hadoop-aws
Steve Loughran 81d90fd65b
HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995)
This patch migrates the S3A connector to use the V2 AWS SDK.

This is a significant change at the source code level.
Any applications using the internal extension/override points in
the filesystem connector are likely to break.

This includes but is not limited to:
- Code invoking methods on the S3AFileSystem class
  which used classes from the V1 SDK.
- The ability to define the factory for the `AmazonS3` client, and
  to retrieve it from the S3AFileSystem. There is a new factory
  API and a special interface S3AInternals to access a limited
  set of internal classes and operations.
- Delegation token and auditing extensions.
- Classes trying to integrate with the AWS SDK.

All standard V1 credential providers listed in the option 
fs.s3a.aws.credentials.provider will be automatically remapped to their
V2 equivalent.

Other V1 Credential Providers are supported, but only if the V1 SDK is
added back to the classpath.  

The SDK Signing plugin has changed; all v1 signers are incompatible.
There is no support for the S3 "v2" signing algorithm.

Finally, the aws-sdk-bundle JAR has been replaced by the shaded V2
equivalent, "bundle.jar", which is now exported by the hadoop-aws module.

Consult the document aws_sdk_upgrade for the full details.

Contributed by Ahmar Suhail + some bits by Steve Loughran
2023-09-11 14:30:25 +01:00
..
dev-support HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995) 2023-09-11 14:30:25 +01:00
src HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995) 2023-09-11 14:30:25 +01:00
pom.xml HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995) 2023-09-11 14:30:25 +01:00