big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event at COMMIT_PENDING state. Contributed by Zhijie Shen

Browse Source 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603338 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Jian He 2014-06-18 00:51:39 +00:00
parent 2298f3d241
commit 0abddac476
3 changed files with 21 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-mapreduce-project/CHANGES.txt
View File

@ -264,6 +264,9 @@ Release 2.5.0 - UNRELEASED
MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth) MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth)
MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event
at COMMIT_PENDING state. (Zhijie Shen via jianhe)
Release 2.4.1 - 2014-06-23 Release 2.4.1 - 2014-06-23
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

9
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
View File

@ -335,6 +335,15 @@ TaskAttemptEventType.TA_CONTAINER_CLEANED, new KilledTransition())
.addTransition(TaskAttemptStateInternal.COMMIT_PENDING, .addTransition(TaskAttemptStateInternal.COMMIT_PENDING,
TaskAttemptStateInternal.FAIL_CONTAINER_CLEANUP, TaskAttemptStateInternal.FAIL_CONTAINER_CLEANUP,
TaskAttemptEventType.TA_TIMED_OUT, CLEANUP_CONTAINER_TRANSITION) TaskAttemptEventType.TA_TIMED_OUT, CLEANUP_CONTAINER_TRANSITION)
// AM is likely to receive duplicate TA_COMMIT_PENDINGs as the task attempt
// will re-send the commit message until it doesn't encounter any
// IOException and succeeds in delivering the commit message.
// Ignoring the duplicate commit message is a short-term fix. In long term,
// we need to make use of retry cache to help this and other MR protocol
// APIs that can be considered as @AtMostOnce.
.addTransition(TaskAttemptStateInternal.COMMIT_PENDING,
TaskAttemptStateInternal.COMMIT_PENDING,
TaskAttemptEventType.TA_COMMIT_PENDING)
// Transitions from SUCCESS_CONTAINER_CLEANUP state // Transitions from SUCCESS_CONTAINER_CLEANUP state
// kill and cleanup the container // kill and cleanup the container

9
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
View File

@ -112,6 +112,15 @@ public void testCommitPending() throws Exception {
//wait for first attempt to commit pending //wait for first attempt to commit pending
app.waitForState(attempt, TaskAttemptState.COMMIT_PENDING); app.waitForState(attempt, TaskAttemptState.COMMIT_PENDING);
//re-send the commit pending signal to the task
app.getContext().getEventHandler().handle(
new TaskAttemptEvent(
attempt.getID(),
TaskAttemptEventType.TA_COMMIT_PENDING));
//the task attempt should be still at COMMIT_PENDING
app.waitForState(attempt, TaskAttemptState.COMMIT_PENDING);
//send the done signal to the task //send the done signal to the task
app.getContext().getEventHandler().handle( app.getContext().getEventHandler().handle(
new TaskAttemptEvent( new TaskAttemptEvent(