big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDFS-9794. Streamer threads may leak if failure happens when closing the striped outputstream. Contributed by Jing Zhao.

Browse Source
This commit is contained in:
Jing Zhao 2016-02-12 10:59:31 -08:00
parent b21bbe9ed1
commit f3c91a41a5
2 changed files with 36 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
View File

@ -822,7 +822,7 @@ protected void closeThreads(boolean force) throws IOException {
streamer.closeSocket();
} catch (Exception e) {
try {
handleCurrentStreamerFailure("force=" + force, e);
handleStreamerFailure("force=" + force, e, streamer);
} catch (IOException ioe) {
b.add(ioe);
}
@ -946,6 +946,7 @@ protected synchronized void closeImpl() throws IOException {
return;
}
try {
try {
// flush from all upper layers
flushBuffer();
@ -977,8 +978,14 @@ protected synchronized void closeImpl() throws IOException {
}
}
}
} finally {
// Failures may happen when flushing data/parity data out. Exceptions
// may be thrown if more than 3 streamers fail, or updatePipeline RPC
// fails. Streamers may keep waiting for the new block/GS information.
// Thus need to force closing these threads.
closeThreads(true);
}
closeThreads(false);
try (TraceScope ignored =
dfsClient.getTracer().newScope("completeFile")) {
completeFile(currentBlockGroup);

3
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -426,6 +426,9 @@ Trunk (Unreleased)
HDFS-9789. Correctly update DataNode's scheduled block size when writing
small EC file. (jing9)
HDFS-9794. Streamer threads may leak if failure happens when closing the
striped outputstream. (jing9)
BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
HDFS-7347. Configurable erasure coding policy for individual files and