hadoop/hadoop-common-project
Mehakmeet Singh aee975a136
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706)
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console.

* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

Contributed by Mehakmeet Singh

Change-Id: Ie1a27a036a39db66a67e9c6d33bc78d54ea708a0
2021-10-05 11:37:41 +01:00
..
hadoop-annotations HADOOP-17947. Provide alternative to Guava VisibleForTesting (#3505) 2021-10-05 10:01:07 +09:00
hadoop-auth YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. Contributed by Tamas Domok 2021-08-24 14:16:15 +02:00
hadoop-auth-examples Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-common HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706) 2021-10-05 11:37:41 +01:00
hadoop-kms HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 2021-07-09 22:50:18 +09:00
hadoop-minikdc Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-nfs HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 2021-08-24 22:44:11 +09:00
hadoop-registry Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
pom.xml Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00