hadoop

History

Allen Wittenauer 730bc746f9 HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw) This commit contains the following JIRA issues: HADOOP-12931. bin/hadoop work for dynamic subcommands HADOOP-12932. bin/yarn work for dynamic subcommands HADOOP-12933. bin/hdfs work for dynamic subcommands HADOOP-12934. bin/mapred work for dynamic subcommands HADOOP-12935. API documentation for dynamic subcommands HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands HADOOP-13086. enable daemonization of dynamic commands HADOOP-13087. env var doc update for dynamic commands HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement HADOOP-13089. hadoop distcp adds client opts twice when dynamic HADOOP-13094. hadoop-common unit tests for dynamic commands HADOOP-13095. hadoop-hdfs unit tests for dynamic commands HADOOP-13107. clean up how rumen is executed HADOOP-13108. dynamic subcommands need a way to manipulate arguments HADOOP-13110. add a streaming subcommand to mapred HADOOP-13111. convert hadoop gridmix to be dynamic HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow HADOOP-13117. clarify daemonization and security vars for dynamic commands HADOOP-13120. add a --debug message when dynamic commands have been used HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop HADOOP-13129. fix typo in dynamic subcommand docs HADOOP-13151. Underscores should be escaped in dynamic subcommands document HADOOP-13153. fix typo in debug statement for dynamic subcommands		2016-05-16 17:54:45 -07:00
..
bin	HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)	2016-05-16 17:54:45 -07:00
conf	HADOOP-11460. Deprecate shell vars (John Smith via aw)	2015-02-04 16:35:50 -08:00
dev-support	YARN-4617. LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy. Contributed by Rohith Sharma K S	2016-01-29 12:22:23 -08:00
hadoop-yarn-api	YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh)	2016-05-13 13:38:36 -07:00
hadoop-yarn-applications	HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.	2016-05-12 18:27:28 -07:00
hadoop-yarn-client	YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or	2016-05-14 12:19:23 -07:00
hadoop-yarn-common	HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.	2016-05-12 18:27:28 -07:00
hadoop-yarn-registry	HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.	2016-05-12 18:27:28 -07:00
hadoop-yarn-server	YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger.	2016-05-16 20:28:04 +00:00
hadoop-yarn-site	HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.	2016-05-12 18:27:28 -07:00
shellprofile.d	HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)	2016-05-16 17:54:45 -07:00
pom.xml	HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.	2016-05-12 18:27:28 -07:00
README

README

YARN (YET ANOTHER RESOURCE NEGOTIATOR or YARN Application Resource Negotiator)
------------------------------------------------------------------------------

Requirements
-------------
Java: JDK 1.6
Maven: Maven 3

Setup
-----
Install protobuf 2.5.0 (Download from http://code.google.com/p/protobuf/downloads/list)
- install the protoc executable (configure, make, make install)
- install the maven artifact (cd java; mvn install)

Quick Maven Tips
----------------
clean workspace: mvn clean
compile and test: mvn install
skip tests: mvn install -DskipTests
skip test execution but compile: mvn install -Dmaven.test.skip.exec=true
clean and test: mvn clean install
run selected test after compile: mvn test -Dtest=TestClassName (combined: mvn clean install -Dtest=TestClassName)
create runnable binaries after install: mvn assembly:assembly -Pnative (combined: mvn clean install assembly:assembly -Pnative)

Eclipse Projects
----------------
http://maven.apache.org/guides/mini/guide-ide-eclipse.html

1. Generate .project and .classpath files in all maven modules
mvn eclipse:eclipse
CAUTION: If the project structure has changed from your previous workspace, clean up all .project and .classpath files recursively. Then run:
mvn eclipse:eclipse

2. Import the projects in eclipse.

3. Set the environment variable M2_REPO to point to your .m2/repository location.

NetBeans Projects
-----------------

NetBeans has builtin support of maven projects. Just "Open Project..."
and everything is setup automatically. Verified with NetBeans 6.9.1.

Custom Hadoop Dependencies
--------------------------

By default Hadoop dependencies are specified in the top-level pom.xml
properties section. One can override them via -Dhadoop-common.version=...
on the command line. ~/.m2/settings.xml can also be used to specify
these properties in different profiles, which is useful for IDEs.

Modules
-------
YARN consists of multiple modules. The modules are listed below as per the directory structure:

hadoop-yarn-api - Yarn's cross platform external interface

hadoop-yarn-common - Utilities which can be used by yarn clients and server

hadoop-yarn-server - Implementation of the hadoop-yarn-api
hadoop-yarn-server-common - APIs shared between resourcemanager and nodemanager
hadoop-yarn-server-nodemanager (TaskTracker replacement)
hadoop-yarn-server-resourcemanager (JobTracker replacement)

Utilities for understanding the code
------------------------------------
Almost all of the yarn components as well as the mapreduce framework use
state-machines for all the data objects. To understand those central pieces of
the code, a visual representation of the state-machines helps much. You can first
convert the state-machines into graphviz(.gv) format by
running:
mvn compile -Pvisualize
Then you can use the dot program for generating directed graphs and convert the above
.gv files to images. The graphviz package has the needed dot program and related
utilites.For e.g., to generate png files you can run:
dot -Tpng NodeManager.gv > NodeManager.png