MAPREDUCE-5184. Document compatibility for MapReduce applications in hadoop-2 vis-a-vis hadoop-1. Contributed by Zhijie Shen.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1493570 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Arun Murthy
parent
98d97d316c
commit
7f96128e43
@ -288,6 +288,9 @@ Release 2.1.0-beta - UNRELEASED
|
|||||||
MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents.
|
MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents.
|
||||||
(cdouglas via acmurthy)
|
(cdouglas via acmurthy)
|
||||||
|
|
||||||
|
MAPREDUCE-5184. Document compatibility for MapReduce applications in
|
||||||
|
hadoop-2 vis-a-vis hadoop-1. (Zhijie Shen via acmurthy)
|
||||||
|
|
||||||
OPTIMIZATIONS
|
OPTIMIZATIONS
|
||||||
|
|
||||||
MAPREDUCE-4974. Optimising the LineRecordReader initialize() method
|
MAPREDUCE-4974. Optimising the LineRecordReader initialize() method
|
||||||
|
|||||||
@ -0,0 +1,107 @@
|
|||||||
|
~~ Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
~~ you may not use this file except in compliance with the License.
|
||||||
|
~~ You may obtain a copy of the License at
|
||||||
|
~~
|
||||||
|
~~ http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
~~
|
||||||
|
~~ Unless required by applicable law or agreed to in writing, software
|
||||||
|
~~ distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
~~ See the License for the specific language governing permissions and
|
||||||
|
~~ limitations under the License. See accompanying LICENSE file.
|
||||||
|
|
||||||
|
---
|
||||||
|
Hadoop Map Reduce Next Generation-${project.version} - Backward Compatibility
|
||||||
|
---
|
||||||
|
---
|
||||||
|
${maven.build.timestamp}
|
||||||
|
|
||||||
|
Apache Hadoop MapReduce - Migrating from Apache Hadoop 1.x to Apache Hadoop 2.x
|
||||||
|
|
||||||
|
\[ {{{../../hadoop-yarn/hadoop-yarn-site/index.html}Go Back}} \]
|
||||||
|
|
||||||
|
* {Introduction}
|
||||||
|
|
||||||
|
This document provides information for users to migrate their Apache Hadoop
|
||||||
|
MapReduce applications from Apache Hadoop 1.x to Apache Hadoop 2.x.
|
||||||
|
|
||||||
|
In Apache Hadoop 2.x we have spun off resource management capabilities
|
||||||
|
into Apache Hadoop YARN, a general purpose, distributed application management
|
||||||
|
framework while Apache Hadoop MapReduce (aka MRv2) remains as a pure
|
||||||
|
distributed computation framework.
|
||||||
|
|
||||||
|
In general, the previous MapReduce runtime (aka MRv1) has been reused and
|
||||||
|
no major surgery has been conducted on it. Therefore, MRv2 is able to ensure
|
||||||
|
satisfactory compatibility with MRv1 applications. However, due to some
|
||||||
|
improvements and code refactorings, a few APIs have been rendered
|
||||||
|
backward-incompatible.
|
||||||
|
|
||||||
|
The remainder of this page will discuss the scope and the level of backward
|
||||||
|
compatibility that we support in Apache Hadoop MapReduce 2.x (MRv2).
|
||||||
|
|
||||||
|
* {Binary Compatibility}
|
||||||
|
|
||||||
|
First, we ensure binary compatibility to the applications that use old
|
||||||
|
<<mapred>> APIs. This means that applications which were built against MRv1
|
||||||
|
<<mapred>> APIs can run directly on YARN without recompilation, merely by
|
||||||
|
pointing them to an Apache Hadoop 2.x cluster via configuration.
|
||||||
|
|
||||||
|
* {Source Compatibility}
|
||||||
|
|
||||||
|
We cannot ensure complete binary compatibility with the applications that use
|
||||||
|
<<mapreduce>> APIs, as these APIs have evolved a lot since MRv1. However, we
|
||||||
|
ensure source compatibility for <<mapreduce>> APIs that break binary
|
||||||
|
compatibility. In other words, users should recompile their applications that
|
||||||
|
use <<mapreduce>> APIs against MRv2 jars. One notable binary incompatibility
|
||||||
|
break is Counter and CounterGroup.
|
||||||
|
|
||||||
|
* {Not Supported}
|
||||||
|
|
||||||
|
MRAdmin has been removed in MRv2 because because <<<mradmin>>> commands
|
||||||
|
no longer exist. They have been replaced by the commands in <<<rmadmin>>>. We
|
||||||
|
neither support binary compatibility nor source compatibility for the
|
||||||
|
applications that use this class directly.
|
||||||
|
|
||||||
|
* {Tradeoffs between MRv1 Users and Early MRv2 Adopters}
|
||||||
|
|
||||||
|
Unfortunately, maintaining binary compatibility for MRv1 applications may lead
|
||||||
|
to binary incompatibility issues for early MRv2 adopters, in particular Hadoop
|
||||||
|
0.23 users. For <<mapred>> APIs, we have chosen to be compatible with MRv1
|
||||||
|
applications, which have a larger user base. For <<mapreduce>> APIs, if they
|
||||||
|
don't significantly break Hadoop 0.23 applications, we still change them to be
|
||||||
|
compatible with MRv1 applications. Below is the list of MapReduce APIs which
|
||||||
|
are incompatible with Hadoop 0.23.
|
||||||
|
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<Problematic Function>> | <<Incompatibility Issue>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.util.ProgramDriver#drive>>> | Return type changes from <<<void>>> to <<<int>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapred.jobcontrol.Job#getMapredJobID>>> | Return type changes from <<<String>>> to <<<JobID>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapred.TaskReport#getTaskId>>> | Return type changes from <<<String>>> to <<<TaskID>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapred.ClusterStatus#UNINITIALIZED_MEMORY_VALUE>>> | Data type changes from <<<long>>> to <<<int>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapreduce.filecache.DistributedCache#getArchiveTimestamps>>> | Return type changes from <<<long[]>>> to <<<String[]>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapreduce.filecache.DistributedCache#getFileTimestamps>>> | Return type changes from <<<long[]>>> to <<<String[]>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapreduce.Job#failTask>>> | Return type changes from <<<void>>> to <<<boolean>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapreduce.Job#killTask>>> | Return type changes from <<<void>>> to <<<boolean>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
| <<<org.apache.hadoop.mapreduce.Job#getTaskCompletionEvents>>> | Return type changes from <<<o.a.h.mapred.TaskCompletionEvent[]>>> to <<<o.a.h.mapreduce.TaskCompletionEvent[]>>> |
|
||||||
|
*-----------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------+
|
||||||
|
|
||||||
|
* {Malicious}
|
||||||
|
|
||||||
|
For the users who are going to try <<<hadoop-examples-1.x.x.jar>>> on YARN,
|
||||||
|
please note that <<<hadoop -jar hadoop-examples-1.x.x.jar>>> will still use
|
||||||
|
<<<hadoop-mapreduce-examples-2.x.x.jar>>>, which is installed together with
|
||||||
|
other MRv2 jars. By default Hadoop framework jars appear before the users'
|
||||||
|
jars in the classpath, such that the classes from the 2.x.x jar will still be
|
||||||
|
picked. Users should either remove <<<hadoop-mapreduce-examples-2.x.x.jar>>>
|
||||||
|
from the classpath or set <<<HADOOP_USER_CLASSPATH_FIRST=true>>> and
|
||||||
|
<<<HADOOP_CLASSPATH=...:hadoop-examples-1.x.x.jar>>> to run their target
|
||||||
|
examples jar.
|
||||||
@ -79,6 +79,7 @@
|
|||||||
</menu>
|
</menu>
|
||||||
|
|
||||||
<menu name="MapReduce" inherit="top">
|
<menu name="MapReduce" inherit="top">
|
||||||
|
<item name="Compatibilty between Hadoop 1.x and Hadoop 2.x" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html"/>
|
||||||
<item name="Encrypted Shuffle" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html"/>
|
<item name="Encrypted Shuffle" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html"/>
|
||||||
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
|
<item name="Pluggable Shuffle/Sort" href="hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html"/>
|
||||||
</menu>
|
</menu>
|
||||||
|
|||||||
@ -53,5 +53,6 @@ MapReduce NextGen aka YARN aka MRv2
|
|||||||
|
|
||||||
* {{{../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html}CLI MiniCluster}}
|
* {{{../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html}CLI MiniCluster}}
|
||||||
|
|
||||||
* {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html}Encrypted Shuffle}}
|
* {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html}Backward Compatibility between Apache Hadoop 1.x and 2.x for MapReduce}}
|
||||||
|
|
||||||
|
* {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html}Encrypted Shuffle}}
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user