big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-4278. Cannot run two local jobs in parallel from the same gateway. Contributed by Sandy Ryza.

Browse Source 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1430363 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Thomas White 2013-01-08 16:26:33 +00:00
parent 9385dd50c7
commit 0f1f5491bc
3 changed files with 12 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-mapreduce-project/CHANGES.txt
View File

@ -238,6 +238,9 @@ Release 2.0.3-alpha - Unreleased
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
TestJobCleanup. (Sandy Ryza via tomwhite) TestJobCleanup. (Sandy Ryza via tomwhite)
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
gateway. (Sandy Ryza via tomwhite)
Release 2.0.2-alpha - 2012-09-07 Release 2.0.2-alpha - 2012-09-07
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

11
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
View File

@ -609,8 +609,12 @@ public LocalJobRunner(JobConf conf) throws IOException {
// JobSubmissionProtocol methods // JobSubmissionProtocol methods
private static int jobid = 0; private static int jobid = 0;
// used for making sure that local jobs run in different jvms don't
// collide on staging or job directories
private int randid;
public synchronized org.apache.hadoop.mapreduce.JobID getNewJobID() { public synchronized org.apache.hadoop.mapreduce.JobID getNewJobID() {
return new org.apache.hadoop.mapreduce.JobID("local", ++jobid); return new org.apache.hadoop.mapreduce.JobID("local" + randid, ++jobid);
} }
public org.apache.hadoop.mapreduce.JobStatus submitJob( public org.apache.hadoop.mapreduce.JobStatus submitJob(
@ -739,10 +743,11 @@ public String getStagingAreaDir() throws IOException {
"/tmp/hadoop/mapred/staging")); "/tmp/hadoop/mapred/staging"));
UserGroupInformation ugi = UserGroupInformation.getCurrentUser(); UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String user; String user;
randid = rand.nextInt(Integer.MAX_VALUE);
if (ugi != null) { if (ugi != null) {
user = ugi.getShortUserName() + rand.nextInt(); user = ugi.getShortUserName() + randid;
} else { } else {
user = "dummy" + rand.nextInt(); user = "dummy" + randid;
} }
return fs.makeQualified(new Path(stagingRootDir, user+"/.staging")).toString(); return fs.makeQualified(new Path(stagingRootDir, user+"/.staging")).toString();
} }

2
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobID.java
View File

@ -32,7 +32,7 @@
* the job. JobID consists of two parts. First part * the job. JobID consists of two parts. First part
* represents the jobtracker identifier, so that jobID to jobtracker map * represents the jobtracker identifier, so that jobID to jobtracker map
* is defined. For cluster setup this string is the jobtracker * is defined. For cluster setup this string is the jobtracker
* start time, for local setting, it is "local". * start time, for local setting, it is "local" and a random number.
* Second part of the JobID is the job number. <br> * Second part of the JobID is the job number. <br>
* An example JobID is : * An example JobID is :
* <code>job_200707121733_0003</code> , which represents the third job * <code>job_200707121733_0003</code> , which represents the third job