hadoop

History

Allen Wittenauer 58ed4fa544 HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS This commit includes the following changes: HADOOP-13356. Add a function to handle command_subcommand_OPTS HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts HADOOP-13357. Modify common to use hadoop_subcommand_opts HADOOP-13359. Modify YARN to use hadoop_subcommand_opts HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity) HADOOP-13564. modify mapred to use hadoop_subcommand_opts HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug HADOOP-13360. Documentation for HADOOP_subcommand_OPTS This closes apache/hadoop#126		2016-09-12 11:10:00 -07:00
..
dev-support	YARN-1021. Yarn Scheduler Load Simulator. (ywskycn via tucu)	2013-09-27 20:23:19 +00:00
src	HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS	2016-09-12 11:10:00 -07:00
pom.xml	Preparing for 3.0.0-alpha2 development	2016-07-15 19:04:17 -07:00
README	YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha)	2015-02-03 13:20:41 -08:00

README

Yarn Scheduler Load Simulator (SLS)

SLS is a stress and performance harness for the Yarn Resource Manager Scheduler
that exercises the scheduler implementation simulating the cluster size and the
applications load without having to have a cluster nor applications.

SLS runs a regular RM without RPC endpoints and uses a NodeManager and
Application Manager simulators to send and receive events simulating cluster
and application load behavior.

==== Quick Start ====

Let $HADOOP_ROOT represent the Hadoop install directory. If you build Hadoop
yourself, $HADOOP_ROOT is hadoop-dist/target/hadoop-$VERSION. The simulator 
is located at $HADOOP_ROOT/share/hadoop/tools/sls. The folder sls contains 
four directories: bin (running scripts), html (web portal to view progress),
sample-conf (some example configurations), and sample-data (an example rumen
trace).

STEP 1: Copy all configuration files (under sample-conf) to $HADOOP_ROOT/etc/hadoop.
STEP 2: Go to the $HADOOP_ROOT/share/hadoop/tools/sls directory, and run the simulator 
using the sample rumen trace (under sample-data).

bin/slsrun.sh —-input-rumen=sample-data/2jobs2min-rumen-jh.json —-output-dir=sample-output

The simulator will start to run, and you can track the running progress 
using its web portal (http://$HOST:10001/simulate, where $HOST is the place 
where you run the simulator.). All collected scheduler metrics are stored 
under the output-dir during running. This trace takes about 3 mins to finish.

For more detailed setup, you can check out the document 
(http://issues.apache.org/jira/secure/attachment/12604817/YARN-1021.pdf)