big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)

Browse Source
This commit is contained in:
Allen Wittenauer 2015-08-06 10:13:18 -07:00
parent e73a928a63
commit fb1be0b310
3 changed files with 29 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-mapreduce-project/CHANGES.txt
View File

@ -206,6 +206,8 @@ Trunk (Unreleased)
MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
to match mapred-default.xml. (Ray Chiang via devaraj) to match mapred-default.xml. (Ray Chiang via devaraj)
MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by

20
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
View File

@ -1760,4 +1760,24 @@
</description> </description>
</property> </property>
<property>
<name>mapreduce.job.encrypted-intermediate-data</name>
<value>false</value>
<description>Encrypt intermediate MapReduce spill files or not
default is false</description>
</property>
<property>
<name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
<value>128</value>
<description>Mapreduce encrypt data key size default is 128</description>
</property>
<property>
<name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
<value>128</value>
<description>Buffer size for intermediate encrypt data in kb
default is 128</description>
</property>
</configuration> </configuration>

8
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
View File

@ -260,4 +260,10 @@ Encrypted Intermediate Data Spill files
This capability allows encryption of the intermediate files generated during the merge and shuffle phases. This capability allows encryption of the intermediate files generated during the merge and shuffle phases.
It can be enabled by setting the `mapreduce.job.encrypted-intermediate-data` job property to `true`. It can be enabled by setting the `mapreduce.job.encrypted-intermediate-data` job property to `true`.
**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number of attempts of the job to 1. | Name | Type | Description |
|:---- |:---- |:---- |
| mapreduce.job.encrypted-intermediate-data | boolean | Enable or disable encrypt intermediate mapreduce spill files.Default is false. |
| mapreduce.job.encrypted-intermediate-data-key-size-bits | int | The key length used by keygenerator to encrypt data spilled to disk. |
| mapreduce.job.encrypted-intermediate-data.buffer.kb | int | The buffer size in kb for stream written to disk after encryption. |
**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number of attempts of the job to 1.