big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn. (thomas graves via mahadev)

Browse Source 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1159421 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Mahadev Konar 2011-08-18 21:25:45 +00:00
parent a027cbf7ce
commit fe87d6a085
3 changed files with 60 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-mapreduce/CHANGES.txt
View File

@ -218,6 +218,9 @@ Trunk (unreleased changes)
MAPREDUCE-2489. Jobsplits with random hostnames can make the queue MAPREDUCE-2489. Jobsplits with random hostnames can make the queue
unusable (jeffrey naisbit via mahadev) unusable (jeffrey naisbit via mahadev)
MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn.
(thomas graves via mahadev)
OPTIMIZATIONS OPTIMIZATIONS
MAPREDUCE-2026. Make JobTracker.getJobCounters() and MAPREDUCE-2026. Make JobTracker.getJobCounters() and

91
hadoop-mapreduce/INSTALL
View File

@ -2,17 +2,17 @@ To compile Hadoop Mapreduce next following, do the following:
Step 1) Install dependencies for yarn Step 1) Install dependencies for yarn
See http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/yarn/README See http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/README
Make sure protbuf library is in your library path or set: export LD_LIBRARY_PATH=/usr/local/lib Make sure protbuf library is in your library path or set: export LD_LIBRARY_PATH=/usr/local/lib
Step 2) Checkout Step 2) Checkout
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/ svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk
Step 3) Build common Step 3) Build common
Go to common directory Go to common directory - choose your regular common build command
ant veryclean mvn-install Example: mvn clean install package -Pbintar -DskipTests
Step 4) Build HDFS Step 4) Build HDFS
@ -24,51 +24,37 @@ Step 5) Build yarn and mapreduce
Go to mapreduce directory Go to mapreduce directory
export MAVEN_OPTS=-Xmx512m export MAVEN_OPTS=-Xmx512m
mvn clean install assembly:assembly
ant veryclean jar jar-test -Dresolvers=internal
In case you want to skip the tests run:
mvn clean install assembly:assembly -DskipTests mvn clean install assembly:assembly -DskipTests
ant veryclean jar jar-test -Dresolvers=internal
Copy in build.properties if appropriate - make sure eclipse.home not set
ant veryclean tar -Dresolvers=internal
You will see a tarball in You will see a tarball in
ls target/hadoop-mapreduce-1.0-SNAPSHOT-bin.tar.gz ls target/hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz
Step 6) Untar the tarball in a clean and different directory. Step 6) Untar the tarball in a clean and different directory.
say HADOOP_YARN_INSTALL say YARN_HOME.
To run Hadoop Mapreduce next applications : Make sure you aren't picking up avro-1.3.2.jar, remove:
$HADOOP_COMMON_HOME/share/hadoop/common/lib/avro-1.3.2.jar
$YARN_HOME/lib/avro-1.3.2.jar
Step 7) cd $HADOOP_YARN_INSTALL Step 7)
Install hdfs/common and start hdfs
Step 8) export the following variables: To run Hadoop Mapreduce next applications:
HADOOP_MAPRED_HOME= Step 8) export the following variables to where you have things installed:
HADOOP_COMMON_HOME= You probably want to export these in hadoop-env.sh and yarn-env.sh also.
HADOOP_HDFS_HOME=
YARN_HOME=directory where you untarred yarn
HADOOP_CONF_DIR=
YARN_CONF_DIR=$HADOOP_CONF_DIR
Step 9) bin/yarn-daemon.sh start resourcemanager export HADOOP_MAPRED_HOME=<mapred loc>
export HADOOP_COMMON_HOME=<common loc>
export HADOOP_HDFS_HOME=<hdfs loc>
export YARN_HOME=directory where you untarred yarn
export HADOOP_CONF_DIR=<conf loc>
export YARN_CONF_DIR=$HADOOP_CONF_DIR
Step 10) bin/yarn-daemon.sh start nodemanager Step 9) Setup config: for running mapreduce applications, which now are in user land, you need to setup nodemanager with the following configuration in your yarn-site.xml before you start the nodemanager.
Step 11) bin/yarn-daemon.sh start historyserver
Step 12) Create the following symlinks in hadoop-common/lib
ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/yarn-api-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/yarn-common-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/yarn-server-common-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/lib/protobuf-java-2.4.0a.jar .
Step 13) Yarn daemons are up! But for running mapreduce applications, which now are in user land, you need to setup nodemanager with the following configuration in your yarn-site.xml before you start the nodemanager.
<property> <property>
<name>nodemanager.auxiluary.services</name> <name>nodemanager.auxiluary.services</name>
<value>mapreduce.shuffle</value> <value>mapreduce.shuffle</value>
@ -79,11 +65,34 @@ Step 13) Yarn daemons are up! But for running mapreduce applications, which now
<value>org.apache.hadoop.mapred.ShuffleHandler</value> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property> </property>
Step 14) You are all set, an example on how to run a mapreduce job is: Step 10) Modify mapred-site.xml to use yarn framework
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
Step 11) Create the following symlinks in $HADOOP_COMMON_HOME/share/hadoop/common/lib
ln -s $YARN_HOME/modules/hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-yarn-api-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-mapreduce-client-common-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-yarn-common-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-mapreduce-client-core-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-yarn-server-common-1.0-SNAPSHOT.jar .
ln -s $YARN_HOME/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar .
Step 12) cd $YARN_HOME
Step 13) bin/yarn-daemon.sh start resourcemanager
Step 14) bin/yarn-daemon.sh start nodemanager
Step 15) bin/yarn-daemon.sh start historyserver
Step 16) You are all set, an example on how to run a mapreduce job is:
cd $HADOOP_MAPRED_HOME cd $HADOOP_MAPRED_HOME
ant examples -Dresolvers=internal ant examples -Dresolvers=internal
$HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars $HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output $HADOOP_COMMON_HOME/bin/hadoop jar $HADOOP_MAPRED_HOME/build/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars $YARN_HOME/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar output
The output on the command line should be almost similar to what you see in the JT/TT setup (Hadoop 0.20/0.21) The output on the command line should be almost similar to what you see in the JT/TT setup (Hadoop 0.20/0.21)

14
hadoop-mapreduce/hadoop-yarn/README
View File

@ -4,7 +4,7 @@ YARN (YET ANOTHER RESOURCE NEGOTIATOR or YARN Application Resource Negotiator)
Requirements Requirements
------------- -------------
Java: JDK 1.6 Java: JDK 1.6
Maven: Maven 2 Maven: Maven 3
Setup Setup
----- -----
@ -63,11 +63,11 @@ Modules
------- -------
YARN consists of multiple modules. The modules are listed below as per the directory structure: YARN consists of multiple modules. The modules are listed below as per the directory structure:
yarn-api - Yarn's cross platform external interface hadoop-yarn-api - Yarn's cross platform external interface
yarn-common - Utilities which can be used by yarn clients and server hadoop-yarn-common - Utilities which can be used by yarn clients and server
yarn-server - Implementation of the yarn-api hadoop-yarn-server - Implementation of the hadoop-yarn-api
yarn-server-common - APIs shared between resourcemanager and nodemanager hadoop-yarn-server-common - APIs shared between resourcemanager and nodemanager
yarn-server-nodemanager (TaskTracker replacement) hadoop-yarn-server-nodemanager (TaskTracker replacement)
yarn-server-resourcemanager (JobTracker replacement) hadoop-yarn-server-resourcemanager (JobTracker replacement)