big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-6923. Optimize MapReduce Shuffle I/O for small partitions. Contributed by Robert Schmidtke.

Browse Source
This commit is contained in:
Ravi Prakash 2017-08-09 15:39:52 -07:00
parent b5c02f95b5
commit ac7d0604bc
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
View File

@ -111,7 +111,10 @@ long customShuffleTransfer(WritableByteChannel target, long position)
long trans = actualCount; long trans = actualCount;
int readSize; int readSize;
ByteBuffer byteBuffer = ByteBuffer.allocate(this.shuffleBufferSize); ByteBuffer byteBuffer = ByteBuffer.allocate(
Math.min(
this.shuffleBufferSize,
trans > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) trans));
while(trans > 0L && while(trans > 0L &&
(readSize = fileChannel.read(byteBuffer, this.position+position)) > 0) { (readSize = fileChannel.read(byteBuffer, this.position+position)) > 0) {