MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev
This commit is contained in:
Xuan
parent
7fa9ea85d4
commit
fc470840a0
@ -1,308 +1,5 @@
|
||||
Hadoop MapReduce Change Log
|
||||
|
||||
Trunk (Unreleased)
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
|
||||
MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically.
|
||||
(Gera Shegalov and Karthik Kambatla via gera)
|
||||
|
||||
MAPREDUCE-5653. DistCp does not honour config-overrides for
|
||||
mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw)
|
||||
|
||||
MAPREDUCE-4424. 'mapred job -list' command should show the job name
|
||||
as well. (Avinash Kujur via aajisaka)
|
||||
|
||||
MAPREDUCE-6336. Enable v2 FileOutputCommitter by default.
|
||||
(Siqi Li via gera)
|
||||
|
||||
NEW FEATURES
|
||||
|
||||
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
|
||||
|
||||
MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation.
|
||||
(Plamen Jeliazkov via shv)
|
||||
|
||||
MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions
|
||||
with poor implementations of Object#hashCode(). (Radim Kolar via cutting)
|
||||
|
||||
MAPREDUCE-5232. Add a configuration to be able to log classpath and other
|
||||
system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv)
|
||||
|
||||
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
|
||||
RM-restart. (Rohith via jianhe)
|
||||
|
||||
MAPREDUCE-2841. Add a native implementation of MapOutputCollector.
|
||||
(see section below for detailed breakdown)
|
||||
|
||||
IMPROVEMENTS
|
||||
|
||||
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
|
||||
|
||||
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
|
||||
job-history related information.
|
||||
|
||||
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
|
||||
(Vinay Thota via amarrk)
|
||||
|
||||
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
|
||||
(Vinay Thota via amarrk)
|
||||
|
||||
MAPREDUCE-2836. Provide option to fail jobs when submitted to non-existent
|
||||
fair scheduler pools. (Ahmed Radwan via todd)
|
||||
|
||||
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
|
||||
native. (tucu)
|
||||
|
||||
MAPREDUCE-3149. Add a test to verify that TokenCache handles file system
|
||||
uri with no authority. (John George via jitendra)
|
||||
|
||||
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
|
||||
client APIs cross MR1 and MR2 (Ahmed via tucu)
|
||||
|
||||
MAPREDUCE-2944. Improve checking of input for JobClient.displayTasks()
|
||||
(XieXianshan via harsh)
|
||||
|
||||
MAPREDUCE-3956. Remove the use of the deprecated Syncable.sync() method from
|
||||
TeraOutputFormat in the terasort example. (szetszwo)
|
||||
|
||||
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
|
||||
(tomwhite)
|
||||
|
||||
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
|
||||
|
||||
MAPREDUCE-3302. Remove the last dependency call from
|
||||
org.apache.hadoop.record package in MR. (harsh)
|
||||
|
||||
MAPREDUCE-2384. The job submitter should make sure to validate
|
||||
jobs before creation of necessary files. (harsh)
|
||||
|
||||
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG
|
||||
(madhukara phatak via bobby)
|
||||
|
||||
MAPREDUCE-4686. hadoop-mapreduce-client-core fails compilation in Eclipse
|
||||
due to missing Avro-generated classes (Chris Nauroth via harsh)
|
||||
|
||||
MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive.
|
||||
(Brandon Li via suresh)
|
||||
|
||||
MAPREDUCE-5197. Add a service for checkpointing task state.
|
||||
(Carlo Curino via cdouglas)
|
||||
|
||||
MAPREDUCE-5189. Add policies and wiring to respond to preemption requests
|
||||
from YARN. (Carlo Curino via cdouglas)
|
||||
|
||||
MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state.
|
||||
(Carlo Curino via cdouglas)
|
||||
|
||||
MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after
|
||||
MAPREDUCE-5196. (Remus Rusanu via cnauroth)
|
||||
|
||||
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
|
||||
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
|
||||
|
||||
MAPREDUCE-6013. [post-HADOOP-9902] mapred version is missing (Akira AJISAKA
|
||||
via aw)
|
||||
|
||||
MAPREDUCE-6250. deprecate sbin/mr-jobhistory-daemon.sh (aw)
|
||||
|
||||
MAPREDUCE-6260. Convert site documentation to markdown (Masatake Iwasaki
|
||||
via aw)
|
||||
|
||||
MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml
|
||||
(Ray Chiang via aw)
|
||||
|
||||
MAPREDUCE-2632. Avoid calling the partitioner when the numReduceTasks is 1.
|
||||
(Ravi Teja Ch N V and Sunil G via kasha)
|
||||
|
||||
MAPREDUCE-6407. Migrate MAPREDUCE nativetask build to new CMake framework
|
||||
(Alan Burlison via Colin P. McCabe)
|
||||
|
||||
MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API
|
||||
(Junping Du via jianhe)
|
||||
|
||||
BUG FIXES
|
||||
|
||||
MAPREDUCE-6191. Improve clearing stale state of Java serialization
|
||||
testcase. (Sam Liu via Eric Yang)
|
||||
|
||||
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
|
||||
(Jinghui Wang via Eric Yang)
|
||||
|
||||
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
|
||||
(Jason Lowe via bobby)
|
||||
|
||||
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
|
||||
(Ravi Prakash and Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks.
|
||||
(Devaraj K and Amar Kamat via amarrk)
|
||||
|
||||
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
|
||||
|
||||
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null.
|
||||
(amarrk)
|
||||
|
||||
MAPREDUCE-2950. [Gridmix] TestUserResolve fails in trunk.
|
||||
(Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
|
||||
ResourceUsageMatcher. (amarrk)
|
||||
|
||||
MAPREDUCE-2978. Fixed test-patch to make Jenkins report correct number of
|
||||
findBugs, correct links to findBugs artifacts and no links to the
|
||||
artifacts when there are no warnings. (Tom White via vinodkv).
|
||||
|
||||
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
|
||||
(Brandon Li via jitendra)
|
||||
|
||||
MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are
|
||||
variable depth (ahmed via tucu) [IMPORTANT: this is dead code in trunk]
|
||||
|
||||
MAPREDUCE-3990. MRBench allows Long-sized input-lines value
|
||||
but parses CLI argument as an Integer. (harsh)
|
||||
|
||||
MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen)
|
||||
|
||||
MAPREDUCE-4685. DBCount should not use ACCESS. (Viji via harsh)
|
||||
|
||||
MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml (tlipcon via harsh)
|
||||
|
||||
MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it
|
||||
(harsh)
|
||||
|
||||
MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes
|
||||
(Joe Mudd via aw)
|
||||
|
||||
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with
|
||||
SerialUtils.hh (Simone Leo via aw)
|
||||
|
||||
MAPREDUCE-4574. Fix TotalOrderParitioner to work with
|
||||
non-WritableComparable key types. (harsh)
|
||||
|
||||
MAPREDUCE-5012. Typo in javadoc for IdentityMapper class. (Adam Monsen
|
||||
via suresh)
|
||||
|
||||
MAPREDUCE-4987. TestMRJobs#testDistributedCache fails on Windows due to
|
||||
classpath problems and unexpected behavior of symlinks (Chris Nauroth via
|
||||
bikas)
|
||||
|
||||
MAPREDUCE-5191. TestQueue#testQueue fails with timeout on Windows. (Ivan
|
||||
Mitic via hitesh)
|
||||
|
||||
MAPREDUCE-5717. Task pings are interpreted as task progress (jlowe)
|
||||
|
||||
MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to
|
||||
ProportionalCapacityPreemptionPolicy (Sunil G via devaraj)
|
||||
|
||||
MAPREDUCE-5972. Fix typo 'programatically' in job.xml (and a few other
|
||||
places) (Akira AJISAKA via aw)
|
||||
|
||||
MAPREDUCE-6161. mapred hsadmin command missing from trunk (Allen Wittenauer
|
||||
via jlowe)
|
||||
|
||||
MAPREDUCE-4413. MR lib dir contains jdiff (which is gpl) (Nemon Lou via aw)
|
||||
|
||||
MAPREDUCE-6234. TestHighRamJob fails due to the change in MAPREDUCE-5785.
|
||||
(Masatake Iwasaki via kasha)
|
||||
|
||||
MAPREDUCE-6343. JobConf.parseMaximumHeapSizeMB() fails to parse value
|
||||
greater than 2GB expressed in bytes. (Hao Xia via kasha)
|
||||
|
||||
MAPREDUCE-6396. TestPipeApplication fails by NullPointerException.
|
||||
(Brahma Reddy Battula via aajisaka)
|
||||
|
||||
MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
|
||||
to match mapred-default.xml. (Ray Chiang via devaraj)
|
||||
|
||||
MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
|
||||
|
||||
MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME
|
||||
(Alan Burlison via aw)
|
||||
|
||||
MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean
|
||||
(Alan Burlison via aw)
|
||||
|
||||
MAPREDUCE-6416. Not all platforms have d_type in struct dirent
|
||||
(Alan Burlison via aw)
|
||||
|
||||
MAPREDUCE-6435. MapReduce client assumes the world is x86
|
||||
(Alan Burlison via aw)
|
||||
|
||||
MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
|
||||
|
||||
MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa)
|
||||
|
||||
BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
|
||||
|
||||
MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by
|
||||
Binglin Chang
|
||||
|
||||
MAPREDUCE-5994. Simplify ByteUtils and fix failing test. (todd)
|
||||
|
||||
MAPREDUCE-5996. native-task: Rename system tests into standard directory
|
||||
layout (todd)
|
||||
|
||||
MAPREDUCE-5997. native-task: Use DirectBufferPool from Hadoop Common (todd)
|
||||
|
||||
MAPREDUCE-6000. native-task: Simplify ByteBufferDataReader/Writer (todd)
|
||||
|
||||
MAPREDUCE-5991. native-task should not run unit tests if native profile is
|
||||
not enabled. (Binglin Chang)
|
||||
|
||||
MAPREDUCE-5995. native-task: Revert changes to Text internals (todd)
|
||||
|
||||
MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang)
|
||||
|
||||
MAPREDUCE-5984. native-task: Reuse lz4 sources in hadoop-common (Binglin
|
||||
Chang)
|
||||
|
||||
MAPREDUCE-5976. native-task: should not fail to build if snappy is missing
|
||||
(Manu Zhang)
|
||||
|
||||
MAPREDUCE-5978. native-task: remove test case for not supported codec
|
||||
Bzip2Codec and DefaultCodec (Manu Zhang)
|
||||
|
||||
MAPREDUCE-6006. native-task: add native tests to maven and fix bug in
|
||||
pom.xml (Binglin Chang via todd)
|
||||
|
||||
MAPREDUCE-6026. native-task: fix logging (Manu Zhang via todd)
|
||||
|
||||
MAPREDUCE-6035. native-task: sources/test-sources jar distribution (Manu
|
||||
Zhang via todd)
|
||||
|
||||
MAPREDUCE-5977. Fix or suppress native-task gcc warnings (Manu Zhang via
|
||||
todd)
|
||||
|
||||
MAPREDUCE-6054. native-task: Speed up tests (todd)
|
||||
|
||||
MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is
|
||||
sucessful (Binglin Chang)
|
||||
|
||||
MAPREDUCE-6056. native-task: move system test working dir to target dir and
|
||||
cleanup test config xml files (Manu Zhang via bchang)
|
||||
|
||||
MAPREDUCE-6055. native-task: findbugs, interface annotations, and other misc
|
||||
cleanup (todd)
|
||||
|
||||
MAPREDUCE-6067. native-task: fix some counter issues (Binglin Chang)
|
||||
|
||||
MAPREDUCE-6069. native-task: Lint/style fixes and removal of unused code
|
||||
(todd)
|
||||
|
||||
MAPREDUCE-6074. native-task: fix release audit, javadoc, javac warnings
|
||||
(todd)
|
||||
|
||||
MAPREDUCE-6077. native-task: Remove CustomModule examples in nativetask (seanzhong)
|
||||
|
||||
MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang)
|
||||
|
||||
MAPREDUCE-5801. Uber mode's log message is missing a vcore reason
|
||||
(Steven Wong via aw)
|
||||
|
||||
MAPREDUCE-6525. Fix test failure of TestMiniMRClientCluster.testRestart.
|
||||
(Masatake Iwasaki via aajisaka)
|
||||
|
||||
Release 2.9.0 - UNRELEASED
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
@ -424,6 +121,9 @@ Release 2.8.0 - UNRELEASED
|
||||
MAPREDUCE-6384. Add the last reporting reducer info for too many fetch
|
||||
failure diagnostics (Chang Li via jlowe)
|
||||
|
||||
MAPREDUCE-5762. Port MAPREDUCE-3223 and MAPREDUCE-4695 (Remove MRv1 config
|
||||
from mapred-default.xml) to branch-2. (aajisaka)
|
||||
|
||||
MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. (Robert Kanter
|
||||
via junping_du)
|
||||
|
||||
@ -445,6 +145,9 @@ Release 2.8.0 - UNRELEASED
|
||||
MAPREDUCE-5763. Warn message about httpshuffle in NM logs.
|
||||
(Akira AJISAKA via ozawa)
|
||||
|
||||
MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter
|
||||
API. Contributed by Junping Du
|
||||
|
||||
MAPREDUCE-6499. Add elapsed time for retired job in JobHistoryServer WebUI.
|
||||
(Lin Yiqun via aajisaka)
|
||||
|
||||
@ -563,6 +266,9 @@ Release 2.8.0 - UNRELEASED
|
||||
MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview.
|
||||
(Siqi Li via gera)
|
||||
|
||||
MAPREDUCE-5232. Add a configuration to be able to log classpath and other
|
||||
system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv)
|
||||
|
||||
MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes
|
||||
(Ray Chiang via ozawa).
|
||||
|
||||
@ -656,6 +362,8 @@ Release 2.8.0 - UNRELEASED
|
||||
MAPREDUCE-6533. testDetermineCacheVisibilities of
|
||||
TestClientDistributedCacheManager is broken (Chang Li via jlowe)
|
||||
|
||||
MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
|
||||
|
||||
MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration.
|
||||
(Gabor Liptak via aajisaka)
|
||||
|
||||
@ -672,6 +380,7 @@ Release 2.7.3 - UNRELEASED
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
|
||||
|
||||
NEW FEATURES
|
||||
|
||||
IMPROVEMENTS
|
||||
@ -747,6 +456,8 @@ Release 2.7.2 - UNRELEASED
|
||||
MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when
|
||||
strategy is dynamic (Kuhu Shukla via kihwal)
|
||||
|
||||
MAPREDUCE-6566. Add retry support to mapreduce CLI tool. (Varun Vasudev via xgong)
|
||||
|
||||
Release 2.7.1 - 2015-07-06
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
@ -941,17 +652,14 @@ Release 2.7.0 - 2015-04-20
|
||||
MAPREDUCE-2815. JavaDoc does not generate correctly for
|
||||
MultithreadedMapRunner. (Chris Palmer via aajisaka)
|
||||
|
||||
MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures.
|
||||
(Varun Saxena via kasha)
|
||||
|
||||
MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi
|
||||
via ozawa)
|
||||
|
||||
MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma
|
||||
Reddy Battula via ozawa)
|
||||
|
||||
MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc
|
||||
comments. (Akira AJISAKA and Andrew Purtell via ozawa)
|
||||
MAPREDUCE-5657. [JDK8] Fix Javadoc errors caused by incorrect or illegal
|
||||
tags in doc comments. (Akira AJISAKA and Andrew Purtell via ozawa)
|
||||
|
||||
MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa)
|
||||
|
||||
@ -1081,6 +789,8 @@ Release 2.6.0 - 2014-11-18
|
||||
INCOMPATIBLE CHANGES
|
||||
|
||||
NEW FEATURES
|
||||
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
|
||||
RM-restart. Contributed by Rohith
|
||||
|
||||
MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server.
|
||||
(Robert Kanter via zjshen)
|
||||
@ -1093,6 +803,9 @@ Release 2.6.0 - 2014-11-18
|
||||
MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
|
||||
compatible/incompatible changes (Junping Du via jlowe)
|
||||
|
||||
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
|
||||
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
|
||||
|
||||
MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
|
||||
Koji Noguchi via aw)
|
||||
|
||||
@ -1148,12 +861,6 @@ Release 2.6.0 - 2014-11-18
|
||||
MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
|
||||
(jlowe)
|
||||
|
||||
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
|
||||
(jlowe)
|
||||
|
||||
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
|
||||
defaults (Sangjin Lee via jlowe)
|
||||
|
||||
MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
|
||||
(Akira AJISAKA via aw)
|
||||
|
||||
@ -1163,6 +870,9 @@ Release 2.6.0 - 2014-11-18
|
||||
MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus
|
||||
and getStatus (Akira AJISAKA via aw)
|
||||
|
||||
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
|
||||
(jlowe)
|
||||
|
||||
MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw)
|
||||
|
||||
MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors
|
||||
@ -1176,6 +886,9 @@ Release 2.6.0 - 2014-11-18
|
||||
|
||||
MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
|
||||
|
||||
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
|
||||
defaults (Sangjin Lee via jlowe)
|
||||
|
||||
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
|
||||
the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
|
||||
|
||||
@ -1499,6 +1212,9 @@ Release 2.4.1 - 2014-06-23
|
||||
|
||||
BUG FIXES
|
||||
|
||||
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
|
||||
(Jinghui Wang via Eric Yang)
|
||||
|
||||
MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
|
||||
|
||||
MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
|
||||
@ -1560,9 +1276,6 @@ Release 2.4.0 - 2014-04-07
|
||||
MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
|
||||
jlowe)
|
||||
|
||||
MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level
|
||||
inside the ApplicationMaster log. (Jian He via vinodkv)
|
||||
|
||||
MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
|
||||
Shegalov via jlowe)
|
||||
|
||||
@ -2677,9 +2390,6 @@ Release 2.0.3-alpha - 2013-02-06
|
||||
|
||||
MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
|
||||
|
||||
MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO.
|
||||
(Brandon Li via suresh)
|
||||
|
||||
MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and
|
||||
UNASSIGNED states. (Mayank Bansal via sseth)
|
||||
|
||||
@ -2697,6 +2407,9 @@ Release 2.0.3-alpha - 2013-02-06
|
||||
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
|
||||
TestJobCleanup. (Sandy Ryza via tomwhite)
|
||||
|
||||
MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred.
|
||||
gridmix.TestResourceUsageEmulators (Dennis Y via tgraves)
|
||||
|
||||
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
|
||||
gateway. (Sandy Ryza via tomwhite)
|
||||
|
||||
@ -2768,23 +2481,11 @@ Release 2.0.2-alpha - 2012-09-07
|
||||
MAPREDUCE-3921. MR AM should act on node health status changes.
|
||||
(Bikas Saha via sseth)
|
||||
|
||||
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
|
||||
mapred-default.xml (Rui Kubo via harsh)
|
||||
|
||||
MAPREDUCE-3907. Document entries mapred-default.xml for the
|
||||
jobhistory server. (Eugene Koontz via harsh)
|
||||
|
||||
MAPREDUCE-3906. Fix inconsistency in documentation regarding
|
||||
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
|
||||
|
||||
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
|
||||
is not used. (Gabriel Reid via harsh)
|
||||
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
|
||||
|
||||
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
|
||||
development of new applications. (Bikas Saha via acmurthy)
|
||||
|
||||
MAPREDUCE-4447. Remove aop from cruft from the ant build. (eli)
|
||||
|
||||
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
|
||||
(Todd Lipcon and Siddharth Seth via sseth)
|
||||
|
||||
@ -2830,7 +2531,7 @@ Release 2.0.2-alpha - 2012-09-07
|
||||
classpath (rkanter via tucu)
|
||||
|
||||
MAPREDUCE-4577. HDFS-3672 broke
|
||||
TestCombineFileInputFormat.testMissingBlocks() test. (atm)
|
||||
TestCombineFileInputFormat.testMissingBlocks() test (atm)
|
||||
|
||||
MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
|
||||
|
||||
@ -2946,6 +2647,9 @@ Release 2.0.0-alpha - 05-23-2012
|
||||
MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in
|
||||
MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
|
||||
|
||||
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
|
||||
(tomwhite)
|
||||
|
||||
MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files
|
||||
splitting. (harsh)
|
||||
|
||||
@ -2978,6 +2682,20 @@ Release 2.0.0-alpha - 05-23-2012
|
||||
MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager
|
||||
(tucu)
|
||||
|
||||
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
|
||||
|
||||
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
|
||||
mapred-default.xml (Rui Kubo via harsh)
|
||||
|
||||
MAPREDUCE-3907. Document entries mapred-default.xml for the
|
||||
jobhistory server. (Eugene Koontz via harsh)
|
||||
|
||||
MAPREDUCE-3906. Fix inconsistency in documentation regarding
|
||||
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
|
||||
|
||||
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
|
||||
is not used. (Gabriel Reid via harsh)
|
||||
|
||||
OPTIMIZATIONS
|
||||
|
||||
BUG FIXES
|
||||
@ -3034,9 +2752,6 @@ Release 2.0.0-alpha - 05-23-2012
|
||||
|
||||
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
|
||||
|
||||
MAPREDUCE-4095. TestJobInProgress#testLocality uses a bogus topology.
|
||||
(Colin Patrick McCabe via eli)
|
||||
|
||||
MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
|
||||
|
||||
MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath
|
||||
@ -3851,6 +3566,9 @@ Release 0.23.3
|
||||
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
|
||||
history (Jason Lowe via bobby)
|
||||
|
||||
MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability
|
||||
(Robert Evans via tucu)
|
||||
|
||||
Release 0.23.2 - UNRELEASED
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
@ -3910,6 +3628,7 @@ Release 0.23.2 - UNRELEASED
|
||||
sseth)
|
||||
|
||||
BUG FIXES
|
||||
|
||||
MAPREDUCE-3918 proc_historyserver no longer in command line arguments for
|
||||
HistoryServer (Jon Eagles via bobby)
|
||||
|
||||
@ -3922,9 +3641,6 @@ Release 0.23.2 - UNRELEASED
|
||||
MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas
|
||||
Graves via mahadev)
|
||||
|
||||
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
|
||||
causes jobs to fail (ahmed via tucu).
|
||||
|
||||
MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
|
||||
HTTPS parameters. (todd)
|
||||
|
||||
@ -4038,20 +3754,39 @@ Release 0.23.2 - UNRELEASED
|
||||
|
||||
Release 0.23.1 - 2012-02-17
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
|
||||
NEW FEATURES
|
||||
|
||||
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
|
||||
|
||||
MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
|
||||
|
||||
MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
|
||||
Graves via vinodkv)
|
||||
|
||||
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster
|
||||
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster.
|
||||
(Anupam Seth via mahadev)
|
||||
|
||||
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
|
||||
|
||||
IMPROVEMENTS
|
||||
|
||||
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
|
||||
|
||||
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
|
||||
job-history related information.
|
||||
|
||||
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
|
||||
(Vinay Thota via amarrk)
|
||||
|
||||
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
|
||||
(Ravi Prakash via bobby)
|
||||
|
||||
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
|
||||
causes jobs to fail (ahmed via tucu).
|
||||
|
||||
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
|
||||
(Vinay Thota via amarrk)
|
||||
|
||||
MAPREDUCE-3297. Moved log related components into yarn-common so that
|
||||
HistoryServer and clients can use them without depending on the
|
||||
yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
|
||||
@ -4080,8 +3815,11 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
|
||||
has wrong configuration or permissions. (Hitesh Shah via vinodkv)
|
||||
|
||||
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution.
|
||||
(tucu)
|
||||
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR
|
||||
resolution. (tucu)
|
||||
|
||||
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
|
||||
client APIs cross MR1 and MR2. (Ahmed via tucu)
|
||||
|
||||
MAPREDUCE-3373. Hadoop scripts unconditionally source
|
||||
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
|
||||
@ -4222,6 +3960,22 @@ Release 0.23.1 - 2012-02-17
|
||||
|
||||
BUG FIXES
|
||||
|
||||
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
|
||||
ResourceUsageMatcher. (amarrk)
|
||||
|
||||
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
|
||||
(Jason Lowe via bobby)
|
||||
|
||||
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
|
||||
(Ravi Prakash and Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
|
||||
|
||||
MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null.
|
||||
(amarrk)
|
||||
|
||||
MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
|
||||
and fixed bugs in it. (Devaraj K via vinodkv)
|
||||
|
||||
@ -4246,6 +4000,8 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
|
||||
TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
|
||||
|
||||
MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv)
|
||||
|
||||
MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
|
||||
which per-container connections to NodeManager were lingering long enough
|
||||
to hit the ulimits on number of processes. (vinodkv)
|
||||
@ -4265,6 +4021,9 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
|
||||
by the integration tests. (Hitesh Shah via vinodkv)
|
||||
|
||||
MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot
|
||||
version. (Jonathan Eagles via todd)
|
||||
|
||||
MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
|
||||
|
||||
MAPREDUCE-3447. mapreduce examples not working (mahadev)
|
||||
@ -4285,12 +4044,18 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
|
||||
cut down noise. (acmurthy)
|
||||
|
||||
MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version.
|
||||
(sseth via acmurthy)
|
||||
|
||||
MAPREDUCE-3433. Finding counters by legacy group name returns empty
|
||||
counters. (tomwhite)
|
||||
|
||||
MAPREDUCE-3450. NM port info no longer available in JobHistory.
|
||||
(Siddharth Seth via mahadev)
|
||||
|
||||
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore.
|
||||
(jeagles via tucu)
|
||||
|
||||
MAPREDUCE-3488. Streaming jobs are failing because the main class
|
||||
isnt set in the pom files. (mahadev)
|
||||
|
||||
@ -4311,6 +4076,8 @@ Release 0.23.1 - 2012-02-17
|
||||
|
||||
MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
|
||||
|
||||
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
|
||||
|
||||
MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for
|
||||
$HADOOP_*_HOME (Eric Payne via mahadev)
|
||||
|
||||
@ -4320,11 +4087,14 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as
|
||||
ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
|
||||
|
||||
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
|
||||
classpath from the build machine. (tucu)
|
||||
|
||||
MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
|
||||
(Jonathan Eagles via acmurthy)
|
||||
|
||||
MAPREDUCE-3147. Handle leaf queues with the same name properly. (Ravi Prakash via
|
||||
mahadev)
|
||||
MAPREDUCE-3147. Handle leaf queues with the same name properly.
|
||||
(Ravi Prakash via mahadev)
|
||||
|
||||
MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value
|
||||
for queues (Anupam Seth via mahadev)
|
||||
@ -4354,6 +4124,12 @@ Release 0.23.1 - 2012-02-17
|
||||
in the correct directory to work properly in secure mode. (Hitesh Shah via
|
||||
vinodkv)
|
||||
|
||||
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be
|
||||
added as ivy dependency. (tucu)
|
||||
|
||||
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives
|
||||
dependency. (tucu)
|
||||
|
||||
MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via
|
||||
mahadev)
|
||||
|
||||
@ -4403,6 +4179,9 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
|
||||
yarn daemons can start. (Arun C Murthy via vinodkv)
|
||||
|
||||
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar
|
||||
Kamat and Devaraj K via sseth)
|
||||
|
||||
MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
|
||||
thereby solving race conditions in MR AM shutdown. (vinodkv)
|
||||
|
||||
@ -4421,9 +4200,6 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
|
||||
ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
|
||||
|
||||
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
|
||||
adding a file/archive to the path. (Dick King via tucu)
|
||||
|
||||
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
|
||||
(sseth)
|
||||
|
||||
@ -4438,6 +4214,9 @@ Release 0.23.1 - 2012-02-17
|
||||
|
||||
MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
|
||||
|
||||
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
|
||||
adding a file/archive to the path. (Dick King via tucu)
|
||||
|
||||
MAPREDUCE-3326. Added detailed information about queue's to the
|
||||
CapacityScheduler web-ui. (Jason Lowe via acmurthy)
|
||||
|
||||
@ -4448,29 +4227,6 @@ Release 0.23.1 - 2012-02-17
|
||||
yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
|
||||
Eagles via acmurthy)
|
||||
|
||||
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
|
||||
missing license header. (Hitesh Shah via tucu).
|
||||
|
||||
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
|
||||
|
||||
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
|
||||
|
||||
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
|
||||
by default. (tucu)
|
||||
|
||||
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. (jeagles via tucu)
|
||||
|
||||
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
|
||||
|
||||
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
|
||||
classpath from the build machine. (tucu)
|
||||
|
||||
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be added as
|
||||
ivy dependency. (tucu)
|
||||
|
||||
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives dependency.
|
||||
(tucu)
|
||||
|
||||
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
|
||||
via acmurthy)
|
||||
|
||||
@ -4507,6 +4263,9 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
|
||||
speculating either maps or reduces. (Eric Payne via vinodkv)
|
||||
|
||||
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
|
||||
(Brandon Li via jitendra)
|
||||
|
||||
MAPREDUCE-3649. Job End notification gives an error on calling back.
|
||||
(Ravi Prakash via mahadev)
|
||||
|
||||
@ -4531,6 +4290,9 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
|
||||
(Thomas Graves via mahadev)
|
||||
|
||||
MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch.
|
||||
(mahadev)
|
||||
|
||||
MAPREDUCE-3549. write api documentation for web service apis for RM, NM,
|
||||
mapreduce app master, and job history server (Thomas Graves via mahadev)
|
||||
|
||||
@ -4691,9 +4453,6 @@ Release 0.23.1 - 2012-02-17
|
||||
MAPREDUCE-3697. Support binary compatibility for Counters after
|
||||
MAPREDUCE-901. (mahadev via acmurthy)
|
||||
|
||||
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
|
||||
mahadev)
|
||||
|
||||
MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
|
||||
jobs. (Arpit Gupta via acmurthy)
|
||||
|
||||
@ -4732,9 +4491,6 @@ Release 0.23.1 - 2012-02-17
|
||||
|
||||
MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
|
||||
|
||||
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
|
||||
(Ravi Prakash via bobby)
|
||||
|
||||
MAPREDUCE-3843. Job summary log file found missing on the RM host
|
||||
(Anupam Seth via tgraves)
|
||||
|
||||
@ -5137,6 +4893,12 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the
|
||||
JobHistoryServer. (Siddharth Seth via vinodkv)
|
||||
|
||||
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
|
||||
by default. (tucu)
|
||||
|
||||
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
|
||||
native. (tucu)
|
||||
|
||||
MAPREDUCE-3146. Added a MR specific command line to dump logs for a
|
||||
given TaskAttemptID. (Siddharth Seth via vinodkv)
|
||||
|
||||
@ -6306,6 +6068,9 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root
|
||||
of the install properly. (Xie Xianshan via acmurthy)
|
||||
|
||||
MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via
|
||||
acmurthy)
|
||||
|
||||
MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames.
|
||||
(vinodkv via acmurthy)
|
||||
|
||||
@ -6413,9 +6178,6 @@ Release 0.23.0 - 2011-11-01
|
||||
|
||||
MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev)
|
||||
|
||||
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
|
||||
tests. (vinodkv)
|
||||
|
||||
MAPREDUCE-3162. Separated application-init and container-init event types
|
||||
in NodeManager's Application state machine. (Todd Lipcon via vinodkv)
|
||||
|
||||
@ -6470,8 +6232,6 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application
|
||||
directories during app-init. (todd via acmurthy)
|
||||
|
||||
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
|
||||
|
||||
MAPREDUCE-2746. Yarn servers can't communicate with each other with
|
||||
hadoop.security.authorization set to true (acmurthy via mahadev)
|
||||
|
||||
@ -6487,9 +6247,6 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-3269. Fixed log4j properties to correctly set logging options
|
||||
for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy)
|
||||
|
||||
MAPREDUCE-2977. Fix ResourceManager to renew HDFS delegation tokens for
|
||||
applications. (acmurthy)
|
||||
|
||||
MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM
|
||||
and prints a lots of logs. (vinodkv via mahadev)
|
||||
|
||||
@ -6513,7 +6270,7 @@ Release 0.23.0 - 2011-11-01
|
||||
|
||||
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its
|
||||
containers (process-trees) irrespective of whether the container succeeded,
|
||||
or killed. (Hitesh Shah via vinodkv)
|
||||
or killed. Contributed by Hitesh Shah.
|
||||
|
||||
MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv)
|
||||
|
||||
@ -6530,7 +6287,7 @@ Release 0.23.0 - 2011-11-01
|
||||
process goes down and comes up while job is getting executed.
|
||||
(Eric Payne via mahadev)
|
||||
|
||||
MAPREDUCE-3209. Jenkins reports 160 FindBugs warnings (mahadev)
|
||||
MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev)
|
||||
|
||||
MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly.
|
||||
(Siddharth Seth via acmurthy)
|
||||
@ -6555,12 +6312,28 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing
|
||||
NodeManager to crash. (vinodkv)
|
||||
|
||||
MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev)
|
||||
|
||||
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
|
||||
missing license header. (Hitesh Shah via tucu).
|
||||
|
||||
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
|
||||
|
||||
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
|
||||
tests. (vinodkv)
|
||||
|
||||
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
|
||||
|
||||
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
|
||||
|
||||
MAPREDUCE-3256. Added authorization checks for the protocol between
|
||||
NodeManager and ApplicationMaster. (vinodkv via acmurthy)
|
||||
|
||||
MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a
|
||||
task-scheduling deadlock. (Robert Joseph Evans via vinodkv)
|
||||
|
||||
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
|
||||
|
||||
MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing
|
||||
TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv)
|
||||
|
||||
@ -6588,10 +6361,10 @@ Release 0.23.0 - 2011-11-01
|
||||
MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
|
||||
|
||||
MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying
|
||||
on current history file name format. (Ravi Gummadi)
|
||||
on current history file name format. (Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file
|
||||
names also. (Ravi Gummadi)
|
||||
names also. (Ravi Gummadi via amarrk)
|
||||
|
||||
MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas).
|
||||
|
||||
@ -7217,9 +6990,6 @@ Release 0.22.0 - 2011-11-29
|
||||
MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity
|
||||
scheduler if queue names contain special symbols. (Priyo Mustafi via shv)
|
||||
|
||||
MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
|
||||
(Ming Ma via shv)
|
||||
|
||||
MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via
|
||||
acmurthy)
|
||||
|
||||
@ -7265,6 +7035,9 @@ Release 0.21.1 - Unreleased
|
||||
MAPREDUCE-2127. mapreduce trunk builds are filing on hudson.
|
||||
(Bruno Mahé via eli)
|
||||
|
||||
MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
|
||||
(Ming Ma via shv)
|
||||
|
||||
Release 0.21.0 - 2010-08-13
|
||||
|
||||
INCOMPATIBLE CHANGES
|
||||
|
||||
@ -26,6 +26,7 @@
|
||||
import java.util.HashSet;
|
||||
import java.util.Arrays;
|
||||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
import org.apache.commons.lang.StringUtils;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
@ -43,6 +44,7 @@
|
||||
import org.apache.hadoop.mapreduce.JobID;
|
||||
import org.apache.hadoop.mapreduce.JobPriority;
|
||||
import org.apache.hadoop.mapreduce.JobStatus;
|
||||
import org.apache.hadoop.mapreduce.MRJobConfig;
|
||||
import org.apache.hadoop.mapreduce.TaskAttemptID;
|
||||
import org.apache.hadoop.mapreduce.TaskCompletionEvent;
|
||||
import org.apache.hadoop.mapreduce.TaskReport;
|
||||
@ -268,7 +270,7 @@ public int run(String[] argv) throws Exception {
|
||||
System.out.println("Created job " + job.getJobID());
|
||||
exitCode = 0;
|
||||
} else if (getStatus) {
|
||||
Job job = cluster.getJob(JobID.forName(jobid));
|
||||
Job job = getJob(JobID.forName(jobid));
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else {
|
||||
@ -283,7 +285,7 @@ public int run(String[] argv) throws Exception {
|
||||
exitCode = 0;
|
||||
}
|
||||
} else if (getCounter) {
|
||||
Job job = cluster.getJob(JobID.forName(jobid));
|
||||
Job job = getJob(JobID.forName(jobid));
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else {
|
||||
@ -299,7 +301,7 @@ public int run(String[] argv) throws Exception {
|
||||
}
|
||||
}
|
||||
} else if (killJob) {
|
||||
Job job = cluster.getJob(JobID.forName(jobid));
|
||||
Job job = getJob(JobID.forName(jobid));
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else {
|
||||
@ -323,7 +325,7 @@ public int run(String[] argv) throws Exception {
|
||||
}
|
||||
}
|
||||
} else if (setJobPriority) {
|
||||
Job job = cluster.getJob(JobID.forName(jobid));
|
||||
Job job = getJob(JobID.forName(jobid));
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else {
|
||||
@ -339,7 +341,7 @@ public int run(String[] argv) throws Exception {
|
||||
viewHistory(historyFile, viewAllHistory);
|
||||
exitCode = 0;
|
||||
} else if (listEvents) {
|
||||
listEvents(cluster.getJob(JobID.forName(jobid)), fromEvent, nEvents);
|
||||
listEvents(getJob(JobID.forName(jobid)), fromEvent, nEvents);
|
||||
exitCode = 0;
|
||||
} else if (listJobs) {
|
||||
listJobs(cluster);
|
||||
@ -354,11 +356,11 @@ public int run(String[] argv) throws Exception {
|
||||
listBlacklistedTrackers(cluster);
|
||||
exitCode = 0;
|
||||
} else if (displayTasks) {
|
||||
displayTasks(cluster.getJob(JobID.forName(jobid)), taskType, taskState);
|
||||
displayTasks(getJob(JobID.forName(jobid)), taskType, taskState);
|
||||
exitCode = 0;
|
||||
} else if(killTask) {
|
||||
TaskAttemptID taskID = TaskAttemptID.forName(taskid);
|
||||
Job job = cluster.getJob(taskID.getJobID());
|
||||
Job job = getJob(taskID.getJobID());
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else if (job.killTask(taskID, false)) {
|
||||
@ -370,7 +372,7 @@ public int run(String[] argv) throws Exception {
|
||||
}
|
||||
} else if(failTask) {
|
||||
TaskAttemptID taskID = TaskAttemptID.forName(taskid);
|
||||
Job job = cluster.getJob(taskID.getJobID());
|
||||
Job job = getJob(taskID.getJobID());
|
||||
if (job == null) {
|
||||
System.out.println("Could not find job " + jobid);
|
||||
} else if(job.killTask(taskID, true)) {
|
||||
@ -532,6 +534,29 @@ protected static String getTaskLogURL(TaskAttemptID taskId, String baseUrl) {
|
||||
return (baseUrl + "/tasklog?plaintext=true&attemptid=" + taskId);
|
||||
}
|
||||
|
||||
@VisibleForTesting
|
||||
Job getJob(JobID jobid) throws IOException, InterruptedException {
|
||||
|
||||
int maxRetry = getConf().getInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES,
|
||||
MRJobConfig.DEFAULT_MR_CLIENT_JOB_MAX_RETRIES);
|
||||
long retryInterval = getConf()
|
||||
.getLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL,
|
||||
MRJobConfig.DEFAULT_MR_CLIENT_JOB_RETRY_INTERVAL);
|
||||
Job job = cluster.getJob(jobid);
|
||||
|
||||
for (int i = 0; i < maxRetry; ++i) {
|
||||
if (job != null) {
|
||||
return job;
|
||||
}
|
||||
LOG.info("Could not obtain job info after " + String.valueOf(i + 1)
|
||||
+ " attempt(s). Sleeping for " + String.valueOf(retryInterval / 1000)
|
||||
+ " seconds and retrying.");
|
||||
Thread.sleep(retryInterval);
|
||||
job = cluster.getJob(jobid);
|
||||
}
|
||||
return job;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Dump a list of currently running jobs
|
||||
|
||||
@ -20,14 +20,19 @@
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import java.io.IOException;
|
||||
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
import org.apache.hadoop.mapreduce.Cluster;
|
||||
import org.apache.hadoop.mapreduce.Job;
|
||||
import org.apache.hadoop.mapreduce.JobID;
|
||||
import org.apache.hadoop.mapreduce.MRJobConfig;
|
||||
import org.apache.hadoop.mapreduce.TaskReport;
|
||||
import org.apache.hadoop.mapreduce.TaskType;
|
||||
import org.apache.hadoop.mapreduce.JobPriority;
|
||||
import org.apache.hadoop.mapreduce.JobStatus;
|
||||
import org.apache.hadoop.mapreduce.JobStatus.State;
|
||||
import org.apache.hadoop.util.Time;
|
||||
import org.junit.Assert;
|
||||
import org.junit.Test;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.when;
|
||||
@ -44,7 +49,7 @@ public void testListAttemptIdsWithValidInput() throws Exception {
|
||||
JobID jobId = JobID.forName(jobIdStr);
|
||||
Cluster mockCluster = mock(Cluster.class);
|
||||
Job job = mock(Job.class);
|
||||
CLI cli = spy(new CLI());
|
||||
CLI cli = spy(new CLI(new Configuration()));
|
||||
|
||||
doReturn(mockCluster).when(cli).createCluster();
|
||||
when(job.getTaskReports(TaskType.MAP)).thenReturn(
|
||||
@ -112,7 +117,7 @@ private TaskReport[] getTaskReports(JobID jobId, TaskType type) {
|
||||
@Test
|
||||
public void testJobKIll() throws Exception {
|
||||
Cluster mockCluster = mock(Cluster.class);
|
||||
CLI cli = spy(new CLI());
|
||||
CLI cli = spy(new CLI(new Configuration()));
|
||||
doReturn(mockCluster).when(cli).createCluster();
|
||||
String jobId1 = "job_1234654654_001";
|
||||
String jobId2 = "job_1234654654_002";
|
||||
@ -149,4 +154,26 @@ private Job mockJob(Cluster mockCluster, String jobId, State jobState)
|
||||
when(mockJob.getStatus()).thenReturn(status);
|
||||
return mockJob;
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetJob() throws Exception {
|
||||
Configuration conf = new Configuration();
|
||||
long sleepTime = 100;
|
||||
conf.setLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL, sleepTime);
|
||||
Cluster mockCluster = mock(Cluster.class);
|
||||
JobID jobId1 = JobID.forName("job_1234654654_001");
|
||||
when(mockCluster.getJob(jobId1)).thenReturn(null);
|
||||
|
||||
for (int i = 0; i < 2; ++i) {
|
||||
conf.setInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES, i);
|
||||
CLI cli = spy(new CLI(conf));
|
||||
cli.cluster = mockCluster;
|
||||
doReturn(mockCluster).when(cli).createCluster();
|
||||
long start = Time.monotonicNow();
|
||||
cli.getJob(jobId1);
|
||||
long end = Time.monotonicNow();
|
||||
Assert.assertTrue(end - start > (i * sleepTime));
|
||||
Assert.assertTrue(end - start < ((i + 1) * sleepTime));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user