hadoop/hadoop-tools
Mehakmeet Singh aee975a136
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706)
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console.

* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

Contributed by Mehakmeet Singh

Change-Id: Ie1a27a036a39db66a67e9c6d33bc78d54ea708a0
2021-10-05 11:37:41 +01:00
..
hadoop-aliyun Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-archive-logs Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-archives Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-aws HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706) 2021-10-05 11:37:41 +01:00
hadoop-azure HADOOP-17934. ABFS: Make sure the AbfsHttpOperation is non-null before using it (#3477) 2021-10-04 20:54:39 +01:00
hadoop-azure-datalake Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-datajoin Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-distcp HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 2021-08-02 12:58:37 +01:00
hadoop-dynamometer HDFS-16050. Some dynamometer tests fail. (#3079) 2021-06-07 15:03:06 +09:00
hadoop-extras Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-fs2img Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-gridmix Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-kafka Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-openstack Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-pipes Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-resourceestimator Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-rumen Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-sls HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 2021-06-17 10:00:36 +09:00
hadoop-streaming Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
hadoop-tools-dist Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
pom.xml Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00