diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt index e910cddf19..c424667132 100644 --- a/hadoop-mapreduce-project/CHANGES.txt +++ b/hadoop-mapreduce-project/CHANGES.txt @@ -206,6 +206,8 @@ Trunk (Unreleased) MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT to match mapred-default.xml. (Ray Chiang via devaraj) + MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw) + BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml index ddcd2dfdee..6d205c5861 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml @@ -1760,4 +1760,24 @@ + + mapreduce.job.encrypted-intermediate-data + false + Encrypt intermediate MapReduce spill files or not + default is false + + + + mapreduce.job.encrypted-intermediate-data-key-size-bits + 128 + Mapreduce encrypt data key size default is 128 + + + + mapreduce.job.encrypted-intermediate-data.buffer.kb + 128 + Buffer size for intermediate encrypt data in kb + default is 128 + + diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md index c23be7a992..fddd84f823 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md @@ -260,4 +260,10 @@ Encrypted Intermediate Data Spill files This capability allows encryption of the intermediate files generated during the merge and shuffle phases. It can be enabled by setting the `mapreduce.job.encrypted-intermediate-data` job property to `true`. -**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number of attempts of the job to 1. \ No newline at end of file +| Name | Type | Description | +|:---- |:---- |:---- | +| mapreduce.job.encrypted-intermediate-data | boolean | Enable or disable encrypt intermediate mapreduce spill files.Default is false. | +| mapreduce.job.encrypted-intermediate-data-key-size-bits | int | The key length used by keygenerator to encrypt data spilled to disk. | +| mapreduce.job.encrypted-intermediate-data.buffer.kb | int | The buffer size in kb for stream written to disk after encryption. | + +**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number of attempts of the job to 1.