big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-6812. Change documentation for correct placement of configuration variables: mapreduce.reduce.input.buffer.percent, mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb. Contributed by Chris Douglas.

Browse Source 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1064917 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Konstantin Shvachko 2011-01-28 22:40:42 +00:00
parent c60feead5e
commit 077bc4af3c
3 changed files with 24 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
CHANGES.txt
View File

@ -268,6 +268,11 @@ Release 0.22.0 - Unreleased
HADOOP-7110. Implement chmod with JNI. (todd) HADOOP-7110. Implement chmod with JNI. (todd)
HADOOP-6812. Change documentation for correct placement of configuration
variables: mapreduce.reduce.input.buffer.percent,
mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb
(Chris Douglas via shv)
OPTIMIZATIONS OPTIMIZATIONS
HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..). HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).

25
src/docs/src/documentation/content/xdocs/cluster_setup.xml
View File

@ -666,22 +666,33 @@
</td> </td>
</tr> </tr>
<tr> <tr>
<td>conf/core-site.xml</td> <td>conf/mapred-site.xml</td>
<td>fs.inmemory.size.mb</td> <td>mapreduce.reduce.shuffle.input.buffer.percent</td>
<td>200</td> <td>0.80</td>
<td> <td>
Larger amount of memory allocated for the in-memory Larger amount of memory allocated for merging map output
file-system used to merge map-outputs at the reduces. in memory during the shuffle. Expressed as a fraction of
the total heap.
</td> </td>
</tr> </tr>
<tr> <tr>
<td>conf/core-site.xml</td> <td>conf/mapred-site.xml</td>
<td>mapreduce.reduce.input.buffer.percent</td>
<td>0.80</td>
<td>
Larger amount of memory allocated for retaining map output
in memory during the reduce. Expressed as a fraction of
the total heap.
</td>
</tr>
<tr>
<td>conf/mapred-site.xml</td>
<td>mapreduce.task.io.sort.factor</td> <td>mapreduce.task.io.sort.factor</td>
<td>100</td> <td>100</td>
<td>More streams merged at once while sorting files.</td> <td>More streams merged at once while sorting files.</td>
</tr> </tr>
<tr> <tr>
<td>conf/core-site.xml</td> <td>conf/mapred-site.xml</td>
<td>mapreduce.task.io.sort.mb</td> <td>mapreduce.task.io.sort.mb</td>
<td>200</td> <td>200</td>
<td>Higher memory-limit while sorting data.</td> <td>Higher memory-limit while sorting data.</td>

2
src/docs/src/documentation/content/xdocs/hod_scheduler.xml
View File

@ -250,7 +250,7 @@ info_port = Port number of the HDFS NameNode web UI
<p>For example:</p> <p>For example:</p>
<source> <source>
server-params = mapred.reduce.parallel.copies=20,io.sort.factor=100,io.sort.mb=128,io.file.buffer.size=131072 server-params = mapred.reduce.parallel.copies=20,io.sort.factor=100,io.sort.mb=128,io.file.buffer.size=131072
final-server-params = mapred.child.java.opts=-Xmx512m,dfs.block.size=134217728,fs.inmemory.size.mb=128 final-server-params = mapred.child.java.opts=-Xmx512m,dfs.block.size=134217728
</source> </source>
<p>In order to provide the options from command line, you can use the following syntax:</p> <p>In order to provide the options from command line, you can use the following syntax:</p>
<p>For configuring the MapReduce daemons use:</p> <p>For configuring the MapReduce daemons use:</p>