big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev

Browse Source
This commit is contained in:
Xuan 2015-12-07 14:15:14 -08:00
parent 7fa9ea85d4
commit fc470840a0
3 changed files with 646 additions and 821 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

561
hadoop-mapreduce-project/CHANGES.txt
View File

@ -1,308 +1,5 @@
Hadoop MapReduce Change Log Hadoop MapReduce Change Log
Trunk (Unreleased)
INCOMPATIBLE CHANGES
MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically.
(Gera Shegalov and Karthik Kambatla via gera)
MAPREDUCE-5653. DistCp does not honour config-overrides for
mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw)
MAPREDUCE-4424. 'mapred job -list' command should show the job name
as well. (Avinash Kujur via aajisaka)
MAPREDUCE-6336. Enable v2 FileOutputCommitter by default.
(Siqi Li via gera)
NEW FEATURES
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation.
(Plamen Jeliazkov via shv)
MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions
with poor implementations of Object#hashCode(). (Radim Kolar via cutting)
MAPREDUCE-5232. Add a configuration to be able to log classpath and other
system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv)
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
RM-restart. (Rohith via jianhe)
MAPREDUCE-2841. Add a native implementation of MapOutputCollector.
(see section below for detailed breakdown)
IMPROVEMENTS
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
job-history related information.
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-2836. Provide option to fail jobs when submitted to non-existent
fair scheduler pools. (Ahmed Radwan via todd)
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
native. (tucu)
MAPREDUCE-3149. Add a test to verify that TokenCache handles file system
uri with no authority. (John George via jitendra)
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
client APIs cross MR1 and MR2 (Ahmed via tucu)
MAPREDUCE-2944. Improve checking of input for JobClient.displayTasks()
(XieXianshan via harsh)
MAPREDUCE-3956. Remove the use of the deprecated Syncable.sync() method from
TeraOutputFormat in the terasort example. (szetszwo)
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
(tomwhite)
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
MAPREDUCE-3302. Remove the last dependency call from
org.apache.hadoop.record package in MR. (harsh)
MAPREDUCE-2384. The job submitter should make sure to validate
jobs before creation of necessary files. (harsh)
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG
(madhukara phatak via bobby)
MAPREDUCE-4686. hadoop-mapreduce-client-core fails compilation in Eclipse
due to missing Avro-generated classes (Chris Nauroth via harsh)
MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive.
(Brandon Li via suresh)
MAPREDUCE-5197. Add a service for checkpointing task state.
(Carlo Curino via cdouglas)
MAPREDUCE-5189. Add policies and wiring to respond to preemption requests
from YARN. (Carlo Curino via cdouglas)
MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state.
(Carlo Curino via cdouglas)
MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after
MAPREDUCE-5196. (Remus Rusanu via cnauroth)
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
MAPREDUCE-6013. [post-HADOOP-9902] mapred version is missing (Akira AJISAKA
via aw)
MAPREDUCE-6250. deprecate sbin/mr-jobhistory-daemon.sh (aw)
MAPREDUCE-6260. Convert site documentation to markdown (Masatake Iwasaki
via aw)
MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml
(Ray Chiang via aw)
MAPREDUCE-2632. Avoid calling the partitioner when the numReduceTasks is 1.
(Ravi Teja Ch N V and Sunil G via kasha)
MAPREDUCE-6407. Migrate MAPREDUCE nativetask build to new CMake framework
(Alan Burlison via Colin P. McCabe)
MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API
(Junping Du via jianhe)
BUG FIXES
MAPREDUCE-6191. Improve clearing stale state of Java serialization
testcase. (Sam Liu via Eric Yang)
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
(Jinghui Wang via Eric Yang)
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
(Jason Lowe via bobby)
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
(Ravi Prakash and Ravi Gummadi via amarrk)
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks.
(Devaraj K and Amar Kamat via amarrk)
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null.
(amarrk)
MAPREDUCE-2950. [Gridmix] TestUserResolve fails in trunk.
(Ravi Gummadi via amarrk)
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
ResourceUsageMatcher. (amarrk)
MAPREDUCE-2978. Fixed test-patch to make Jenkins report correct number of
findBugs, correct links to findBugs artifacts and no links to the
artifacts when there are no warnings. (Tom White via vinodkv).
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
(Brandon Li via jitendra)
MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are
variable depth (ahmed via tucu) [IMPORTANT: this is dead code in trunk]
MAPREDUCE-3990. MRBench allows Long-sized input-lines value
but parses CLI argument as an Integer. (harsh)
MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen)
MAPREDUCE-4685. DBCount should not use ACCESS. (Viji via harsh)
MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml (tlipcon via harsh)
MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it
(harsh)
MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes
(Joe Mudd via aw)
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with
SerialUtils.hh (Simone Leo via aw)
MAPREDUCE-4574. Fix TotalOrderParitioner to work with
non-WritableComparable key types. (harsh)
MAPREDUCE-5012. Typo in javadoc for IdentityMapper class. (Adam Monsen
via suresh)
MAPREDUCE-4987. TestMRJobs#testDistributedCache fails on Windows due to
classpath problems and unexpected behavior of symlinks (Chris Nauroth via
bikas)
MAPREDUCE-5191. TestQueue#testQueue fails with timeout on Windows. (Ivan
Mitic via hitesh)
MAPREDUCE-5717. Task pings are interpreted as task progress (jlowe)
MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to
ProportionalCapacityPreemptionPolicy (Sunil G via devaraj)
MAPREDUCE-5972. Fix typo 'programatically' in job.xml (and a few other
places) (Akira AJISAKA via aw)
MAPREDUCE-6161. mapred hsadmin command missing from trunk (Allen Wittenauer
via jlowe)
MAPREDUCE-4413. MR lib dir contains jdiff (which is gpl) (Nemon Lou via aw)
MAPREDUCE-6234. TestHighRamJob fails due to the change in MAPREDUCE-5785.
(Masatake Iwasaki via kasha)
MAPREDUCE-6343. JobConf.parseMaximumHeapSizeMB() fails to parse value
greater than 2GB expressed in bytes. (Hao Xia via kasha)
MAPREDUCE-6396. TestPipeApplication fails by NullPointerException.
(Brahma Reddy Battula via aajisaka)
MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
to match mapred-default.xml. (Ray Chiang via devaraj)
MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME
(Alan Burlison via aw)
MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean
(Alan Burlison via aw)
MAPREDUCE-6416. Not all platforms have d_type in struct dirent
(Alan Burlison via aw)
MAPREDUCE-6435. MapReduce client assumes the world is x86
(Alan Burlison via aw)
MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa)
BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by
Binglin Chang
MAPREDUCE-5994. Simplify ByteUtils and fix failing test. (todd)
MAPREDUCE-5996. native-task: Rename system tests into standard directory
layout (todd)
MAPREDUCE-5997. native-task: Use DirectBufferPool from Hadoop Common (todd)
MAPREDUCE-6000. native-task: Simplify ByteBufferDataReader/Writer (todd)
MAPREDUCE-5991. native-task should not run unit tests if native profile is
not enabled. (Binglin Chang)
MAPREDUCE-5995. native-task: Revert changes to Text internals (todd)
MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang)
MAPREDUCE-5984. native-task: Reuse lz4 sources in hadoop-common (Binglin
Chang)
MAPREDUCE-5976. native-task: should not fail to build if snappy is missing
(Manu Zhang)
MAPREDUCE-5978. native-task: remove test case for not supported codec
Bzip2Codec and DefaultCodec (Manu Zhang)
MAPREDUCE-6006. native-task: add native tests to maven and fix bug in
pom.xml (Binglin Chang via todd)
MAPREDUCE-6026. native-task: fix logging (Manu Zhang via todd)
MAPREDUCE-6035. native-task: sources/test-sources jar distribution (Manu
Zhang via todd)
MAPREDUCE-5977. Fix or suppress native-task gcc warnings (Manu Zhang via
todd)
MAPREDUCE-6054. native-task: Speed up tests (todd)
MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is
sucessful (Binglin Chang)
MAPREDUCE-6056. native-task: move system test working dir to target dir and
cleanup test config xml files (Manu Zhang via bchang)
MAPREDUCE-6055. native-task: findbugs, interface annotations, and other misc
cleanup (todd)
MAPREDUCE-6067. native-task: fix some counter issues (Binglin Chang)
MAPREDUCE-6069. native-task: Lint/style fixes and removal of unused code
(todd)
MAPREDUCE-6074. native-task: fix release audit, javadoc, javac warnings
(todd)
MAPREDUCE-6077. native-task: Remove CustomModule examples in nativetask (seanzhong)
MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang)
MAPREDUCE-5801. Uber mode's log message is missing a vcore reason
(Steven Wong via aw)
MAPREDUCE-6525. Fix test failure of TestMiniMRClientCluster.testRestart.
(Masatake Iwasaki via aajisaka)
Release 2.9.0 - UNRELEASED Release 2.9.0 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -424,6 +121,9 @@ Release 2.8.0 - UNRELEASED
MAPREDUCE-6384. Add the last reporting reducer info for too many fetch MAPREDUCE-6384. Add the last reporting reducer info for too many fetch
failure diagnostics (Chang Li via jlowe) failure diagnostics (Chang Li via jlowe)
MAPREDUCE-5762. Port MAPREDUCE-3223 and MAPREDUCE-4695 (Remove MRv1 config
from mapred-default.xml) to branch-2. (aajisaka)
MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. (Robert Kanter MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. (Robert Kanter
via junping_du) via junping_du)
@ -445,6 +145,9 @@ Release 2.8.0 - UNRELEASED
MAPREDUCE-5763. Warn message about httpshuffle in NM logs. MAPREDUCE-5763. Warn message about httpshuffle in NM logs.
(Akira AJISAKA via ozawa) (Akira AJISAKA via ozawa)
MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter
API. Contributed by Junping Du
MAPREDUCE-6499. Add elapsed time for retired job in JobHistoryServer WebUI. MAPREDUCE-6499. Add elapsed time for retired job in JobHistoryServer WebUI.
(Lin Yiqun via aajisaka) (Lin Yiqun via aajisaka)
@ -563,6 +266,9 @@ Release 2.8.0 - UNRELEASED
MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview.
(Siqi Li via gera) (Siqi Li via gera)
MAPREDUCE-5232. Add a configuration to be able to log classpath and other
system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv)
MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes
(Ray Chiang via ozawa). (Ray Chiang via ozawa).
@ -656,6 +362,8 @@ Release 2.8.0 - UNRELEASED
MAPREDUCE-6533. testDetermineCacheVisibilities of MAPREDUCE-6533. testDetermineCacheVisibilities of
TestClientDistributedCacheManager is broken (Chang Li via jlowe) TestClientDistributedCacheManager is broken (Chang Li via jlowe)
MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration. MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration.
(Gabor Liptak via aajisaka) (Gabor Liptak via aajisaka)
@ -672,6 +380,7 @@ Release 2.7.3 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
NEW FEATURES NEW FEATURES
IMPROVEMENTS IMPROVEMENTS
@ -747,6 +456,8 @@ Release 2.7.2 - UNRELEASED
MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when
strategy is dynamic (Kuhu Shukla via kihwal) strategy is dynamic (Kuhu Shukla via kihwal)
MAPREDUCE-6566. Add retry support to mapreduce CLI tool. (Varun Vasudev via xgong)
Release 2.7.1 - 2015-07-06 Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -941,17 +652,14 @@ Release 2.7.0 - 2015-04-20
MAPREDUCE-2815. JavaDoc does not generate correctly for MAPREDUCE-2815. JavaDoc does not generate correctly for
MultithreadedMapRunner. (Chris Palmer via aajisaka) MultithreadedMapRunner. (Chris Palmer via aajisaka)
MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures.
(Varun Saxena via kasha)
MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi
via ozawa) via ozawa)
MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma
Reddy Battula via ozawa) Reddy Battula via ozawa)
MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc MAPREDUCE-5657. [JDK8] Fix Javadoc errors caused by incorrect or illegal
comments. (Akira AJISAKA and Andrew Purtell via ozawa) tags in doc comments. (Akira AJISAKA and Andrew Purtell via ozawa)
MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa) MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa)
@ -1081,6 +789,8 @@ Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
NEW FEATURES NEW FEATURES
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
RM-restart. Contributed by Rohith
MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server.
(Robert Kanter via zjshen) (Robert Kanter via zjshen)
@ -1093,6 +803,9 @@ Release 2.6.0 - 2014-11-18
MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
compatible/incompatible changes (Junping Du via jlowe) compatible/incompatible changes (Junping Du via jlowe)
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
Koji Noguchi via aw) Koji Noguchi via aw)
@ -1148,12 +861,6 @@ Release 2.6.0 - 2014-11-18
MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
(jlowe) (jlowe)
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
(jlowe)
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
defaults (Sangjin Lee via jlowe)
MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
(Akira AJISAKA via aw) (Akira AJISAKA via aw)
@ -1163,6 +870,9 @@ Release 2.6.0 - 2014-11-18
MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus
and getStatus (Akira AJISAKA via aw) and getStatus (Akira AJISAKA via aw)
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
(jlowe)
MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw) MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw)
MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors
@ -1176,6 +886,9 @@ Release 2.6.0 - 2014-11-18
MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw) MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
defaults (Sangjin Lee via jlowe)
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
the current context's FS is different. (Benjamin Zhitomirsky via zjshen) the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
@ -1499,6 +1212,9 @@ Release 2.4.1 - 2014-06-23
BUG FIXES BUG FIXES
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
(Jinghui Wang via Eric Yang)
MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen) MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
@ -1560,9 +1276,6 @@ Release 2.4.0 - 2014-04-07
MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
jlowe) jlowe)
MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level
inside the ApplicationMaster log. (Jian He via vinodkv)
MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
Shegalov via jlowe) Shegalov via jlowe)
@ -2677,9 +2390,6 @@ Release 2.0.3-alpha - 2013-02-06
MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite) MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO.
(Brandon Li via suresh)
MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and
UNASSIGNED states. (Mayank Bansal via sseth) UNASSIGNED states. (Mayank Bansal via sseth)
@ -2697,6 +2407,9 @@ Release 2.0.3-alpha - 2013-02-06
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
TestJobCleanup. (Sandy Ryza via tomwhite) TestJobCleanup. (Sandy Ryza via tomwhite)
MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred.
gridmix.TestResourceUsageEmulators (Dennis Y via tgraves)
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
gateway. (Sandy Ryza via tomwhite) gateway. (Sandy Ryza via tomwhite)
@ -2768,23 +2481,11 @@ Release 2.0.2-alpha - 2012-09-07
MAPREDUCE-3921. MR AM should act on node health status changes. MAPREDUCE-3921. MR AM should act on node health status changes.
(Bikas Saha via sseth) (Bikas Saha via sseth)
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
mapred-default.xml (Rui Kubo via harsh)
MAPREDUCE-3907. Document entries mapred-default.xml for the
jobhistory server. (Eugene Koontz via harsh)
MAPREDUCE-3906. Fix inconsistency in documentation regarding
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
is not used. (Gabriel Reid via harsh)
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
development of new applications. (Bikas Saha via acmurthy) development of new applications. (Bikas Saha via acmurthy)
MAPREDUCE-4447. Remove aop from cruft from the ant build. (eli)
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
(Todd Lipcon and Siddharth Seth via sseth) (Todd Lipcon and Siddharth Seth via sseth)
@ -2830,7 +2531,7 @@ Release 2.0.2-alpha - 2012-09-07
classpath (rkanter via tucu) classpath (rkanter via tucu)
MAPREDUCE-4577. HDFS-3672 broke MAPREDUCE-4577. HDFS-3672 broke
TestCombineFileInputFormat.testMissingBlocks() test. (atm) TestCombineFileInputFormat.testMissingBlocks() test (atm)
MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu) MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
@ -2946,6 +2647,9 @@ Release 2.0.0-alpha - 05-23-2012
MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in
MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth) MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
(tomwhite)
MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files
splitting. (harsh) splitting. (harsh)
@ -2978,6 +2682,20 @@ Release 2.0.0-alpha - 05-23-2012
MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager
(tucu) (tucu)
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
mapred-default.xml (Rui Kubo via harsh)
MAPREDUCE-3907. Document entries mapred-default.xml for the
jobhistory server. (Eugene Koontz via harsh)
MAPREDUCE-3906. Fix inconsistency in documentation regarding
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
is not used. (Gabriel Reid via harsh)
OPTIMIZATIONS OPTIMIZATIONS
BUG FIXES BUG FIXES
@ -3034,9 +2752,6 @@ Release 2.0.0-alpha - 05-23-2012
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
MAPREDUCE-4095. TestJobInProgress#testLocality uses a bogus topology.
(Colin Patrick McCabe via eli)
MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu) MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath
@ -3851,6 +3566,9 @@ Release 0.23.3
MAPREDUCE-4641. Exception in commitJob marks job as successful in job MAPREDUCE-4641. Exception in commitJob marks job as successful in job
history (Jason Lowe via bobby) history (Jason Lowe via bobby)
MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability
(Robert Evans via tucu)
Release 0.23.2 - UNRELEASED Release 0.23.2 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -3910,6 +3628,7 @@ Release 0.23.2 - UNRELEASED
sseth) sseth)
BUG FIXES BUG FIXES
MAPREDUCE-3918 proc_historyserver no longer in command line arguments for MAPREDUCE-3918 proc_historyserver no longer in command line arguments for
HistoryServer (Jon Eagles via bobby) HistoryServer (Jon Eagles via bobby)
@ -3922,9 +3641,6 @@ Release 0.23.2 - UNRELEASED
MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas
Graves via mahadev) Graves via mahadev)
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
causes jobs to fail (ahmed via tucu).
MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
HTTPS parameters. (todd) HTTPS parameters. (todd)
@ -4038,20 +3754,39 @@ Release 0.23.2 - UNRELEASED
Release 0.23.1 - 2012-02-17 Release 0.23.1 - 2012-02-17
INCOMPATIBLE CHANGES
NEW FEATURES NEW FEATURES
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev) MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
Graves via vinodkv) Graves via vinodkv)
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster.
(Anupam Seth via mahadev) (Anupam Seth via mahadev)
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
IMPROVEMENTS IMPROVEMENTS
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
job-history related information.
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
(Ravi Prakash via bobby)
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
causes jobs to fail (ahmed via tucu).
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
(Vinay Thota via amarrk)
MAPREDUCE-3297. Moved log related components into yarn-common so that MAPREDUCE-3297. Moved log related components into yarn-common so that
HistoryServer and clients can use them without depending on the HistoryServer and clients can use them without depending on the
yarn-server-nodemanager module. (Siddharth Seth via vinodkv) yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
@ -4080,8 +3815,11 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
has wrong configuration or permissions. (Hitesh Shah via vinodkv) has wrong configuration or permissions. (Hitesh Shah via vinodkv)
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution. MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR
(tucu) resolution. (tucu)
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
client APIs cross MR1 and MR2. (Ahmed via tucu)
MAPREDUCE-3373. Hadoop scripts unconditionally source MAPREDUCE-3373. Hadoop scripts unconditionally source
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
@ -4222,6 +3960,22 @@ Release 0.23.1 - 2012-02-17
BUG FIXES BUG FIXES
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
ResourceUsageMatcher. (amarrk)
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
(Jason Lowe via bobby)
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
(Ravi Prakash and Ravi Gummadi via amarrk)
MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk)
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null.
(amarrk)
MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
and fixed bugs in it. (Devaraj K via vinodkv) and fixed bugs in it. (Devaraj K via vinodkv)
@ -4246,6 +4000,8 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv) TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv)
MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
which per-container connections to NodeManager were lingering long enough which per-container connections to NodeManager were lingering long enough
to hit the ulimits on number of processes. (vinodkv) to hit the ulimits on number of processes. (vinodkv)
@ -4265,6 +4021,9 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
by the integration tests. (Hitesh Shah via vinodkv) by the integration tests. (Hitesh Shah via vinodkv)
MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot
version. (Jonathan Eagles via todd)
MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
MAPREDUCE-3447. mapreduce examples not working (mahadev) MAPREDUCE-3447. mapreduce examples not working (mahadev)
@ -4285,12 +4044,18 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
cut down noise. (acmurthy) cut down noise. (acmurthy)
MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version.
(sseth via acmurthy)
MAPREDUCE-3433. Finding counters by legacy group name returns empty MAPREDUCE-3433. Finding counters by legacy group name returns empty
counters. (tomwhite) counters. (tomwhite)
MAPREDUCE-3450. NM port info no longer available in JobHistory. MAPREDUCE-3450. NM port info no longer available in JobHistory.
(Siddharth Seth via mahadev) (Siddharth Seth via mahadev)
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore.
(jeagles via tucu)
MAPREDUCE-3488. Streaming jobs are failing because the main class MAPREDUCE-3488. Streaming jobs are failing because the main class
isnt set in the pom files. (mahadev) isnt set in the pom files. (mahadev)
@ -4311,6 +4076,8 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite) MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for
$HADOOP_*_HOME (Eric Payne via mahadev) $HADOOP_*_HOME (Eric Payne via mahadev)
@ -4320,11 +4087,14 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as
ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev) ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
classpath from the build machine. (tucu)
MAPREDUCE-3496. Fixed client to print queue acls in consistent order. MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
(Jonathan Eagles via acmurthy) (Jonathan Eagles via acmurthy)
MAPREDUCE-3147. Handle leaf queues with the same name properly. (Ravi Prakash via MAPREDUCE-3147. Handle leaf queues with the same name properly.
mahadev) (Ravi Prakash via mahadev)
MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value
for queues (Anupam Seth via mahadev) for queues (Anupam Seth via mahadev)
@ -4354,6 +4124,12 @@ Release 0.23.1 - 2012-02-17
in the correct directory to work properly in secure mode. (Hitesh Shah via in the correct directory to work properly in secure mode. (Hitesh Shah via
vinodkv) vinodkv)
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be
added as ivy dependency. (tucu)
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives
dependency. (tucu)
MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via
mahadev) mahadev)
@ -4403,6 +4179,9 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
yarn daemons can start. (Arun C Murthy via vinodkv) yarn daemons can start. (Arun C Murthy via vinodkv)
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar
Kamat and Devaraj K via sseth)
MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
thereby solving race conditions in MR AM shutdown. (vinodkv) thereby solving race conditions in MR AM shutdown. (vinodkv)
@ -4421,9 +4200,6 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv) ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
adding a file/archive to the path. (Dick King via tucu)
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
(sseth) (sseth)
@ -4438,6 +4214,9 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth) MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
adding a file/archive to the path. (Dick King via tucu)
MAPREDUCE-3326. Added detailed information about queue's to the MAPREDUCE-3326. Added detailed information about queue's to the
CapacityScheduler web-ui. (Jason Lowe via acmurthy) CapacityScheduler web-ui. (Jason Lowe via acmurthy)
@ -4448,29 +4227,6 @@ Release 0.23.1 - 2012-02-17
yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
Eagles via acmurthy) Eagles via acmurthy)
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
missing license header. (Hitesh Shah via tucu).
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
by default. (tucu)
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. (jeagles via tucu)
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
classpath from the build machine. (tucu)
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be added as
ivy dependency. (tucu)
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives dependency.
(tucu)
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
via acmurthy) via acmurthy)
@ -4507,6 +4263,9 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
speculating either maps or reduces. (Eric Payne via vinodkv) speculating either maps or reduces. (Eric Payne via vinodkv)
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
(Brandon Li via jitendra)
MAPREDUCE-3649. Job End notification gives an error on calling back. MAPREDUCE-3649. Job End notification gives an error on calling back.
(Ravi Prakash via mahadev) (Ravi Prakash via mahadev)
@ -4531,6 +4290,9 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3689. RM web UI doesn't handle newline in job name. MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
(Thomas Graves via mahadev) (Thomas Graves via mahadev)
MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch.
(mahadev)
MAPREDUCE-3549. write api documentation for web service apis for RM, NM, MAPREDUCE-3549. write api documentation for web service apis for RM, NM,
mapreduce app master, and job history server (Thomas Graves via mahadev) mapreduce app master, and job history server (Thomas Graves via mahadev)
@ -4691,9 +4453,6 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3697. Support binary compatibility for Counters after MAPREDUCE-3697. Support binary compatibility for Counters after
MAPREDUCE-901. (mahadev via acmurthy) MAPREDUCE-901. (mahadev via acmurthy)
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
mahadev)
MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
jobs. (Arpit Gupta via acmurthy) jobs. (Arpit Gupta via acmurthy)
@ -4732,9 +4491,6 @@ Release 0.23.1 - 2012-02-17
MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk) MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
(Ravi Prakash via bobby)
MAPREDUCE-3843. Job summary log file found missing on the RM host MAPREDUCE-3843. Job summary log file found missing on the RM host
(Anupam Seth via tgraves) (Anupam Seth via tgraves)
@ -5137,6 +4893,12 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the
JobHistoryServer. (Siddharth Seth via vinodkv) JobHistoryServer. (Siddharth Seth via vinodkv)
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
by default. (tucu)
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
native. (tucu)
MAPREDUCE-3146. Added a MR specific command line to dump logs for a MAPREDUCE-3146. Added a MR specific command line to dump logs for a
given TaskAttemptID. (Siddharth Seth via vinodkv) given TaskAttemptID. (Siddharth Seth via vinodkv)
@ -6306,6 +6068,9 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root
of the install properly. (Xie Xianshan via acmurthy) of the install properly. (Xie Xianshan via acmurthy)
MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via
acmurthy)
MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames. MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames.
(vinodkv via acmurthy) (vinodkv via acmurthy)
@ -6413,9 +6178,6 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev) MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev)
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
tests. (vinodkv)
MAPREDUCE-3162. Separated application-init and container-init event types MAPREDUCE-3162. Separated application-init and container-init event types
in NodeManager's Application state machine. (Todd Lipcon via vinodkv) in NodeManager's Application state machine. (Todd Lipcon via vinodkv)
@ -6470,8 +6232,6 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application
directories during app-init. (todd via acmurthy) directories during app-init. (todd via acmurthy)
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
MAPREDUCE-2746. Yarn servers can't communicate with each other with MAPREDUCE-2746. Yarn servers can't communicate with each other with
hadoop.security.authorization set to true (acmurthy via mahadev) hadoop.security.authorization set to true (acmurthy via mahadev)
@ -6487,9 +6247,6 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3269. Fixed log4j properties to correctly set logging options MAPREDUCE-3269. Fixed log4j properties to correctly set logging options
for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy) for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy)
MAPREDUCE-2977. Fix ResourceManager to renew HDFS delegation tokens for
applications. (acmurthy)
MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM
and prints a lots of logs. (vinodkv via mahadev) and prints a lots of logs. (vinodkv via mahadev)
@ -6513,7 +6270,7 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its
containers (process-trees) irrespective of whether the container succeeded, containers (process-trees) irrespective of whether the container succeeded,
or killed. (Hitesh Shah via vinodkv) or killed. Contributed by Hitesh Shah.
MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv) MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv)
@ -6530,7 +6287,7 @@ Release 0.23.0 - 2011-11-01
process goes down and comes up while job is getting executed. process goes down and comes up while job is getting executed.
(Eric Payne via mahadev) (Eric Payne via mahadev)
MAPREDUCE-3209. Jenkins reports 160 FindBugs warnings (mahadev) MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev)
MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly. MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly.
(Siddharth Seth via acmurthy) (Siddharth Seth via acmurthy)
@ -6555,12 +6312,28 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing
NodeManager to crash. (vinodkv) NodeManager to crash. (vinodkv)
MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev)
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
missing license header. (Hitesh Shah via tucu).
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
tests. (vinodkv)
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
MAPREDUCE-3256. Added authorization checks for the protocol between MAPREDUCE-3256. Added authorization checks for the protocol between
NodeManager and ApplicationMaster. (vinodkv via acmurthy) NodeManager and ApplicationMaster. (vinodkv via acmurthy)
MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a
task-scheduling deadlock. (Robert Joseph Evans via vinodkv) task-scheduling deadlock. (Robert Joseph Evans via vinodkv)
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing
TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv) TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv)
@ -6588,10 +6361,10 @@ Release 0.23.0 - 2011-11-01
MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk) MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying
on current history file name format. (Ravi Gummadi) on current history file name format. (Ravi Gummadi via amarrk)
MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file
names also. (Ravi Gummadi) names also. (Ravi Gummadi via amarrk)
MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas). MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas).
@ -7217,9 +6990,6 @@ Release 0.22.0 - 2011-11-29
MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity
scheduler if queue names contain special symbols. (Priyo Mustafi via shv) scheduler if queue names contain special symbols. (Priyo Mustafi via shv)
MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
(Ming Ma via shv)
MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via
acmurthy) acmurthy)
@ -7265,6 +7035,9 @@ Release 0.21.1 - Unreleased
MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. MAPREDUCE-2127. mapreduce trunk builds are filing on hudson.
(Bruno Mahé via eli) (Bruno Mahé via eli)
MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
(Ming Ma via shv)
Release 0.21.0 - 2010-08-13 Release 0.21.0 - 2010-08-13
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

41
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
View File

@ -26,6 +26,7 @@
import java.util.HashSet; import java.util.HashSet;
import java.util.Arrays; import java.util.Arrays;
import com.google.common.annotations.VisibleForTesting;
import org.apache.commons.lang.StringUtils; import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log; import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory; import org.apache.commons.logging.LogFactory;
@ -43,6 +44,7 @@
import org.apache.hadoop.mapreduce.JobID; import org.apache.hadoop.mapreduce.JobID;
import org.apache.hadoop.mapreduce.JobPriority; import org.apache.hadoop.mapreduce.JobPriority;
import org.apache.hadoop.mapreduce.JobStatus; import org.apache.hadoop.mapreduce.JobStatus;
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.TaskAttemptID; import org.apache.hadoop.mapreduce.TaskAttemptID;
import org.apache.hadoop.mapreduce.TaskCompletionEvent; import org.apache.hadoop.mapreduce.TaskCompletionEvent;
import org.apache.hadoop.mapreduce.TaskReport; import org.apache.hadoop.mapreduce.TaskReport;
@ -268,7 +270,7 @@ public int run(String[] argv) throws Exception {
System.out.println("Created job " + job.getJobID()); System.out.println("Created job " + job.getJobID());
exitCode = 0; exitCode = 0;
} else if (getStatus) { } else if (getStatus) {
Job job = cluster.getJob(JobID.forName(jobid)); Job job = getJob(JobID.forName(jobid));
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else { } else {
@ -283,7 +285,7 @@ public int run(String[] argv) throws Exception {
exitCode = 0; exitCode = 0;
} }
} else if (getCounter) { } else if (getCounter) {
Job job = cluster.getJob(JobID.forName(jobid)); Job job = getJob(JobID.forName(jobid));
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else { } else {
@ -299,7 +301,7 @@ public int run(String[] argv) throws Exception {
} }
} }
} else if (killJob) { } else if (killJob) {
Job job = cluster.getJob(JobID.forName(jobid)); Job job = getJob(JobID.forName(jobid));
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else { } else {
@ -323,7 +325,7 @@ public int run(String[] argv) throws Exception {
} }
} }
} else if (setJobPriority) { } else if (setJobPriority) {
Job job = cluster.getJob(JobID.forName(jobid)); Job job = getJob(JobID.forName(jobid));
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else { } else {
@ -339,7 +341,7 @@ public int run(String[] argv) throws Exception {
viewHistory(historyFile, viewAllHistory); viewHistory(historyFile, viewAllHistory);
exitCode = 0; exitCode = 0;
} else if (listEvents) { } else if (listEvents) {
listEvents(cluster.getJob(JobID.forName(jobid)), fromEvent, nEvents); listEvents(getJob(JobID.forName(jobid)), fromEvent, nEvents);
exitCode = 0; exitCode = 0;
} else if (listJobs) { } else if (listJobs) {
listJobs(cluster); listJobs(cluster);
@ -354,11 +356,11 @@ public int run(String[] argv) throws Exception {
listBlacklistedTrackers(cluster); listBlacklistedTrackers(cluster);
exitCode = 0; exitCode = 0;
} else if (displayTasks) { } else if (displayTasks) {
displayTasks(cluster.getJob(JobID.forName(jobid)), taskType, taskState); displayTasks(getJob(JobID.forName(jobid)), taskType, taskState);
exitCode = 0; exitCode = 0;
} else if(killTask) { } else if(killTask) {
TaskAttemptID taskID = TaskAttemptID.forName(taskid); TaskAttemptID taskID = TaskAttemptID.forName(taskid);
Job job = cluster.getJob(taskID.getJobID()); Job job = getJob(taskID.getJobID());
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else if (job.killTask(taskID, false)) { } else if (job.killTask(taskID, false)) {
@ -370,7 +372,7 @@ public int run(String[] argv) throws Exception {
} }
} else if(failTask) { } else if(failTask) {
TaskAttemptID taskID = TaskAttemptID.forName(taskid); TaskAttemptID taskID = TaskAttemptID.forName(taskid);
Job job = cluster.getJob(taskID.getJobID()); Job job = getJob(taskID.getJobID());
if (job == null) { if (job == null) {
System.out.println("Could not find job " + jobid); System.out.println("Could not find job " + jobid);
} else if(job.killTask(taskID, true)) { } else if(job.killTask(taskID, true)) {
@ -532,6 +534,29 @@ protected static String getTaskLogURL(TaskAttemptID taskId, String baseUrl) {
return (baseUrl + "/tasklog?plaintext=true&attemptid=" + taskId); return (baseUrl + "/tasklog?plaintext=true&attemptid=" + taskId);
} }
@VisibleForTesting
Job getJob(JobID jobid) throws IOException, InterruptedException {
int maxRetry = getConf().getInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES,
MRJobConfig.DEFAULT_MR_CLIENT_JOB_MAX_RETRIES);
long retryInterval = getConf()
.getLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL,
MRJobConfig.DEFAULT_MR_CLIENT_JOB_RETRY_INTERVAL);
Job job = cluster.getJob(jobid);
for (int i = 0; i < maxRetry; ++i) {
if (job != null) {
return job;
}
LOG.info("Could not obtain job info after " + String.valueOf(i + 1)
+ " attempt(s). Sleeping for " + String.valueOf(retryInterval / 1000)
+ " seconds and retrying.");
Thread.sleep(retryInterval);
job = cluster.getJob(jobid);
}
return job;
}
/** /**
* Dump a list of currently running jobs * Dump a list of currently running jobs

31
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
View File

@ -20,14 +20,19 @@
import static org.junit.Assert.*; import static org.junit.Assert.*;
import java.io.IOException; import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Cluster; import org.apache.hadoop.mapreduce.Cluster;
import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.JobID; import org.apache.hadoop.mapreduce.JobID;
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.TaskReport; import org.apache.hadoop.mapreduce.TaskReport;
import org.apache.hadoop.mapreduce.TaskType; import org.apache.hadoop.mapreduce.TaskType;
import org.apache.hadoop.mapreduce.JobPriority; import org.apache.hadoop.mapreduce.JobPriority;
import org.apache.hadoop.mapreduce.JobStatus; import org.apache.hadoop.mapreduce.JobStatus;
import org.apache.hadoop.mapreduce.JobStatus.State; import org.apache.hadoop.mapreduce.JobStatus.State;
import org.apache.hadoop.util.Time;
import org.junit.Assert;
import org.junit.Test; import org.junit.Test;
import static org.mockito.Mockito.mock; import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when; import static org.mockito.Mockito.when;
@ -44,7 +49,7 @@ public void testListAttemptIdsWithValidInput() throws Exception {
JobID jobId = JobID.forName(jobIdStr); JobID jobId = JobID.forName(jobIdStr);
Cluster mockCluster = mock(Cluster.class); Cluster mockCluster = mock(Cluster.class);
Job job = mock(Job.class); Job job = mock(Job.class);
CLI cli = spy(new CLI()); CLI cli = spy(new CLI(new Configuration()));
doReturn(mockCluster).when(cli).createCluster(); doReturn(mockCluster).when(cli).createCluster();
when(job.getTaskReports(TaskType.MAP)).thenReturn( when(job.getTaskReports(TaskType.MAP)).thenReturn(
@ -112,7 +117,7 @@ private TaskReport[] getTaskReports(JobID jobId, TaskType type) {
@Test @Test
public void testJobKIll() throws Exception { public void testJobKIll() throws Exception {
Cluster mockCluster = mock(Cluster.class); Cluster mockCluster = mock(Cluster.class);
CLI cli = spy(new CLI()); CLI cli = spy(new CLI(new Configuration()));
doReturn(mockCluster).when(cli).createCluster(); doReturn(mockCluster).when(cli).createCluster();
String jobId1 = "job_1234654654_001"; String jobId1 = "job_1234654654_001";
String jobId2 = "job_1234654654_002"; String jobId2 = "job_1234654654_002";
@ -149,4 +154,26 @@ private Job mockJob(Cluster mockCluster, String jobId, State jobState)
when(mockJob.getStatus()).thenReturn(status); when(mockJob.getStatus()).thenReturn(status);
return mockJob; return mockJob;
} }
@Test
public void testGetJob() throws Exception {
Configuration conf = new Configuration();
long sleepTime = 100;
conf.setLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL, sleepTime);
Cluster mockCluster = mock(Cluster.class);
JobID jobId1 = JobID.forName("job_1234654654_001");
when(mockCluster.getJob(jobId1)).thenReturn(null);
for (int i = 0; i < 2; ++i) {
conf.setInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES, i);
CLI cli = spy(new CLI(conf));
cli.cluster = mockCluster;
doReturn(mockCluster).when(cli).createCluster();
long start = Time.monotonicNow();
cli.getJob(jobId1);
long end = Time.monotonicNow();
Assert.assertTrue(end - start > (i * sleepTime));
Assert.assertTrue(end - start < ((i + 1) * sleepTime));
}
}
} }