big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog. Contributed by Kousuke Saruta.

Browse Source 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598209 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Devarajulu K 2014-05-29 07:05:26 +00:00
parent 342da5b4d3
commit 61c59c0704
2 changed files with 17 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-mapreduce-project/CHANGES.txt
View File

@ -244,6 +244,9 @@ Release 2.5.0 - UNRELEASED
MAPREDUCE-5862. Line records longer than 2x split size aren't handled MAPREDUCE-5862. Line records longer than 2x split size aren't handled
correctly (bc Wong via jlowe) correctly (bc Wong via jlowe)
MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog.
(Kousuke Saruta via devaraj)
Release 2.4.1 - UNRELEASED Release 2.4.1 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

24
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
View File

@ -199,16 +199,18 @@ void writeToIndexFile(String logLocation,
// file first and then rename. // file first and then rename.
File tmpIndexFile = getTmpIndexFile(currentTaskid, isCleanup); File tmpIndexFile = getTmpIndexFile(currentTaskid, isCleanup);
BufferedOutputStream bos = BufferedOutputStream bos = null;
new BufferedOutputStream( DataOutputStream dos = null;
SecureIOUtils.createForWrite(tmpIndexFile, 0644));
DataOutputStream dos = new DataOutputStream(bos);
//the format of the index file is
//LOG_DIR: <the dir where the task logs are really stored>
//STDOUT: <start-offset in the stdout file> <length>
//STDERR: <start-offset in the stderr file> <length>
//SYSLOG: <start-offset in the syslog file> <length>
try{ try{
bos = new BufferedOutputStream(
SecureIOUtils.createForWrite(tmpIndexFile, 0644));
dos = new DataOutputStream(bos);
//the format of the index file is
//LOG_DIR: <the dir where the task logs are really stored>
//STDOUT: <start-offset in the stdout file> <length>
//STDERR: <start-offset in the stderr file> <length>
//SYSLOG: <start-offset in the syslog file> <length>
dos.writeBytes(LogFileDetail.LOCATION + logLocation + "\n" dos.writeBytes(LogFileDetail.LOCATION + logLocation + "\n"
+ LogName.STDOUT.toString() + ":"); + LogName.STDOUT.toString() + ":");
dos.writeBytes(Long.toString(prevOutLength) + " "); dos.writeBytes(Long.toString(prevOutLength) + " ");
@ -225,8 +227,10 @@ void writeToIndexFile(String logLocation,
+ "\n"); + "\n");
dos.close(); dos.close();
dos = null; dos = null;
bos.close();
bos = null;
} finally { } finally {
IOUtils.cleanup(LOG, dos); IOUtils.cleanup(LOG, dos, bos);
} }
File indexFile = getIndexFile(currentTaskid, isCleanup); File indexFile = getIndexFile(currentTaskid, isCleanup);