hadoop/hadoop-tools
Mehakmeet Singh f813554769
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706)
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console. 
  
* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

Contributed by Mehakmeet Singh
2021-07-27 11:08:51 +01:00
..
hadoop-aliyun HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-archive-logs HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-archives HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-aws HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706) 2021-07-27 11:08:51 +01:00
hadoop-azure Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml (#3223) 2021-07-25 13:10:27 +05:30
hadoop-azure-datalake HADOOP-13327 Output Stream Specification. (#2587) 2021-02-10 10:28:59 +00:00
hadoop-datajoin HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-distcp HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 2021-07-10 13:55:11 +05:30
hadoop-dynamometer HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-extras Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-federation-balance HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
hadoop-fs2img HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-gridmix HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 2021-04-06 13:39:10 +09:00
hadoop-kafka HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-openstack HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-rumen HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 2021-04-06 13:39:10 +09:00
hadoop-sls HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 2021-06-17 09:58:47 +09:00
hadoop-streaming HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-tools-dist HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
pom.xml HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00