big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16,551 Commits 2 Branches 2 Tags 264 MiB
71847ed44d
Commit Graph

406 Commits

Author SHA1 Message Date
Akira Ajisaka
69c21d2db0 MAPREDUCE-6641. TestTaskAttempt fails in trunk. Contributed by Haibo Chen. 2016-06-23 13:26:10 +09:00
Tsuyoshi Ozawa
5d58858bb6 HADOOP-9613. [JDK8] Update jersey version to latest 1.x release. 2016-06-21 08:05:32 +09:00
Wangda Tan
c77a1095dc YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:06:38 -07:00
Ming Ma
4a1cedc010 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma) 2016-06-06 14:30:51 -07:00
Varun Vasudev
42f90ab885 YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 21:24:16 +05:30
Akira Ajisaka
3b3b63081b MAPREDUCE-6607. Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set. Contributed by Kai Sasaki. 2016-05-23 07:24:31 +09:00
Jason Lowe
c9bb96fa81 MAPREDUCE-6689. MapReduce job can infinitely increase number of reducer resource requests. Contributed by Wangda Tan 2016-05-06 22:25:47 +00:00
Vinod Kumar Vavilapalli
8d48266720 MAPREDUCE-6514. Fixed MapReduce ApplicationMaster to properly updated resources ask after ramping down of all reducers avoiding job hangs. Contributed by Varun Saxena and Wangda Tan. 2016-05-05 19:03:27 -07:00
Robert Kanter
9d3fcdfbb3 MAPREDUCE-6675. TestJobImpl.testUnusableNode failed (haibochen via rkanter) 2016-05-04 22:42:21 -07:00
Robert Kanter
ecce3b7d53 MAPREDUCE-6677. LocalContainerAllocator doesn't specify resource of the containers allocated (haibochen via rkanter) 2016-05-04 22:19:20 -07:00
Wangda Tan
8b2880c0b6 MAPREDUCE-6513. MR job got hanged forever when one NM unstable for some time. (Varun Saxena via wangda) 2016-04-14 11:00:53 -07:00
Robert Kanter
3be1ab485f MAPREDUCE-6647. MR usage counters use the resources requested instead of the resources allocated (haibochen via rkanter) 2016-04-06 17:15:43 -07:00
Robert Kanter
ff0ee84d77 MAPREDUCE-4785. TestMRApp occasionally fails (haibochen via rkanter) 2016-03-03 16:38:07 -08:00
Akira Ajisaka
5e565ce7a7 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta." 
This reverts commit fe124da5ff.
 2016-02-10 23:43:33 +09:00
Akira Ajisaka
fe124da5ff HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 2016-02-06 19:31:01 +09:00
Karthik Kambatla
d40859fab1 YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha) 2016-01-18 10:58:14 +01:00
Akira Ajisaka
7dafee11d8 MAPREDUCE-6593. TestJobHistoryEventHandler.testTimelineEventHandling fails on trunk because of NPE. Contributed by Naganarasimha G R. 2016-01-04 16:30:55 +09:00
Junping Du
15d577bfbb Tests in mapreduce-client-app are writing outside of target. Contributed by Akira AJISAKA. 2015-11-25 09:15:26 -08:00
Tsuyoshi Ozawa
3e85542a7a MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) 2015-11-26 01:02:27 +09:00
Jason Lowe
f634505d48 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G 2015-11-24 22:07:26 +00:00
Tsuyoshi Ozawa
ce31b22739 YARN-4320. TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188. Contributed by Varun Saxena. 2015-10-30 18:00:20 +09:00
Jason Lowe
cf953b6258 MAPREDUCE-6515. Update Application priority in AM side from AM-RM heartbeat. Contributed by Sunil G 2015-10-29 18:05:01 +00:00
Karthik Kambatla
4aa9b3e75c MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha) 2015-10-09 07:37:39 -07:00
rohithsharmaks
439f43ad3d MAPREDUCE-6485. Create a new task attempt with failed map task priority if in-progress attempts are unassigned. (Xianyin Xin via rohithsharmaks) 2015-10-02 20:34:34 +05:30
Jason Lowe
ab11085b81 MAPREDUCE-6492. AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask. Contributed by Bibin A Chundatt 2015-09-28 22:12:03 +00:00
Jian He
89cab1ba5f YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 2015-09-23 13:29:38 -07:00
Jian He
83a18add10 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) 2015-09-23 13:29:36 -07:00
Jian He
dfe2cb849f YARN-3866. AM-RM protocol changes to support container resizing. Contributed by Meng Ding 2015-09-23 13:29:36 -07:00
Zhihai Xu
e3ace31e2b MAPREDUCE-6460. TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails. Contributed by Zhihai Xu. 2015-09-19 00:10:06 -07:00
Jason Lowe
ee4ee6af6a MAPREDUCE-5982. Task attempts that fail from the ASSIGNED state can disappear. Contributed by Chang Li 2015-09-17 21:37:39 +00:00
Jason Lowe
3f82f582e5 MAPREDUCE-5002. AM could potentially allocate a reduce container to a map attempt. Contributed by Chang Li 2015-09-17 18:17:29 +00:00
Karthik Kambatla
8dfec7a197 MAPREDUCE-6439. AM may fail instead of retrying if RM shuts down during the allocate call. (Anubhav Dhoot via kasha) 2015-08-15 00:52:11 -07:00
Karthik Kambatla
27d24f96ab MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) 2015-08-14 12:29:50 -07:00
Jason Lowe
d0e0ba8010 MAPREDUCE-6394. Speed up Task processing loop in HsTasksBlock#render(). Contributed by Ray Chiang 2015-07-31 18:17:54 +00:00
Zhihai Xu
93d50b7824 MAPREDUCE-6433. launchTime may be negative. Contributed by Zhihai Xu 2015-07-30 23:07:31 -07:00
Jason Lowe
eac1d18943 MAPREDUCE-6418. MRApp should not shutdown LogManager during shutdown. Contributed by Chang Li 2015-07-01 18:01:22 +00:00
Jason Lowe
2ac87df578 MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang 2015-07-01 16:00:03 +00:00
Jason Lowe
b6ba56457c MAPREDUCE-6384. Add the last reporting reducer info for too many fetch failure diagnostics. Contributed by Chang Li 2015-06-30 21:22:30 +00:00
Gera Shegalov
11ac848207 MAPREDUCE-6408. Queue name and user name should be printed on the job page. (Siqi Li via gera) 2015-06-22 14:40:32 -07:00
Gera Shegalov
63d40d5286 MAPREDUCE-6316. Task Attempt List entries should link to the task overview. (Siqi Li via gera) 2015-06-20 11:38:21 -07:00
Tsuyoshi Ozawa
6afe20a7a4 MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes. Contributed by Ray Chiang. 2015-06-08 15:06:10 -07:00
Wangda Tan
3164e7d838 MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) 2015-05-27 14:26:03 -07:00
Vinayakumar B
360dff5903 MAPREDUCE-6360. TestMapreduceConfigFields is placed in wrong dir, introducing compile error (Contributed by Arshad Mohammad) 2015-05-12 12:49:16 +05:30
Jason Lowe
444836b3dc MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma 2015-05-11 22:37:35 +00:00
Karthik Kambatla
1773aac780 MAPREDUCE-6353. Divide by zero error in MR AM when calculating available containers. (Anubhav Dhoot via kasha) 2015-05-09 14:43:18 -07:00
Karthik Kambatla
70fb37cd79 YARN-1287. Consolidate MockClocks. (Sebastian Wong and Anubhav Dhoot via kasha) 2015-05-09 14:34:54 -07:00
Tsuyoshi Ozawa
d18f10ad1b MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. 2015-05-08 15:59:46 +09:00
Jason Lowe
f30065c8b6 MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne 2015-05-07 22:05:12 +00:00
Akira Ajisaka
d335071398 MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 2015-05-07 19:39:24 +09:00
Robert Kanter
9809a16d3c MAPREDUCE-6192. Create unit test to automatically compare MR related classes and mapred-default.xml (rchiang via rkanter) 2015-05-05 14:43:28 -07:00
Jason Lowe
bf70c5ae28 MAPREDUCE-6259. IllegalArgumentException due to missing job submit time. Contributed by zhihai xu 2015-05-04 20:39:18 +00:00
Devaraj K
f5b38477f9 MAPREDUCE-6339. Job history file is not flushed correctly because 
isTimerActive flag is not set true when flushTimerTask is scheduled.
Contributed by zhihai xu.
 2015-04-30 12:33:22 +05:30
Vinod Kumar Vavilapalli
9fc32c5c4d MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. 2015-04-27 14:58:16 -07:00
Gera Shegalov
78c6b46241 MAPREDUCE-6333. TestEvents,TestAMWebServicesTasks,TestAppController are broken due to MAPREDUCE-6297. (Siqi Li via gera) 2015-04-24 17:31:10 -07:00
Hitesh Shah
265ed1fe80 YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 2015-04-08 14:13:10 -07:00
Jason Lowe
30da99cbaf MAPREDUCE-6277. Job can post multiple history files if attempt loses connection to the RM. Contributed by Chang Li 2015-03-18 19:30:37 +00:00
Tsuyoshi Ozawa
9d38520c8e MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers. Contributed by Zhihai Xu 2015-03-14 16:44:02 +09:00
Jonathan Eagles
8180e676ab YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 2015-03-13 12:04:30 -05:00
Junping Du
4228de9402 MAPREDUCE-5583. Ability to limit running map and reduce tasks. Contributed by Jason Lowe. 2015-03-03 02:02:28 -08:00
Akira Ajisaka
0d7d508c82 MAPREDUCE-6264. Remove httpclient dependency from hadoop-mapreduce-client. Contributed by Brahma Reddy Battula. 2015-02-24 11:34:26 -08:00
Robert Kanter
8acc5e9b4b MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) 2015-02-02 13:51:08 -08:00
Jian He
cff05bff1f MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly. Contributed by Jason Lowe 2015-01-28 15:51:30 -08:00
Gera Shegalov
a003f71cac MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically. (Gera Shegalov and Karthik Kambatla via gera) 2015-01-21 18:42:03 -08:00
Jian He
a4f2995b9e Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-12-15 11:04:09 -08:00
Jason Lowe
03ab24aa01 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 2014-12-03 17:02:14 +00:00
Zhijie Shen
51af8d367d YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 2014-12-01 19:04:04 -08:00
Jason Lowe
0c588904f8 MAPREDUCE-6160. Potential NullPointerException in MRClientProtocol interface implementation. Contributed by Rohith 2014-12-01 22:39:22 +00:00
Karthik Kambatla
a655973e78 Revert "MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically. (Gera Shegalov and Karthik Kambatla via kasha)" 
This reverts commit a4df9eed05.
 2014-11-25 16:24:26 -08:00
Karthik Kambatla
a4df9eed05 MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically. (Gera Shegalov and Karthik Kambatla via kasha) 2014-11-21 18:36:41 -08:00
Arun C. Murthy
43cd07b408 YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 2014-11-09 14:57:37 -08:00
Zhijie Shen
ed63b11646 MAPREDUCE-6052. Supported overriding the default container-log4j.properties file per job. Contributed by Junping Du. 2014-11-01 00:47:57 -07:00
Vinod Kumar Vavilapalli
3f1441245d MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and TestMRTimelineEventHandling. Contributed by Zhijie Shen. 2014-10-29 09:57:39 -07:00
Zhijie Shen
6b2f11b54b MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. 2014-10-27 20:35:40 -07:00
Zhijie Shen
0f3b6900be YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 2014-10-23 21:56:03 -07:00
Jian He
0af1a2b5bc YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:22:07 -07:00
Jason Lowe
bbe80cdc7b MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails. Contributed by Mit Desai 2014-10-13 15:08:23 +00:00
Jason Lowe
2e789eb226 MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit Desai 2014-10-07 14:10:39 +00:00
Zhijie Shen
376233cdd4 MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN scheduler resource type is memory plus cpu. Contributed by Peng Zhang and Varun Vasudev. 2014-09-22 09:28:47 -07:00
Zhijie Shen
6fe5c6b746 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 2014-09-18 21:56:56 -07:00
Vinod Kumar Vavilapalli
14e2639fd0 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 2014-09-13 18:04:05 -07:00
Zhijie Shen
524a63e59e MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir creation on Windows. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619863 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 17:06:23 +00:00
Jason Darrell Lowe
d51f81c3b6 MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2. Contributed by chang li 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618691 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 18:18:22 +00:00
Eric Yang
08d6201b34 MAPREDUCE-6033. Updated access check for displaying job information 
(Yu Gao via Eric Yang)



git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618648 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 16:13:56 +00:00
Zhijie Shen
84bc2fe402 MAPREDUCE-6032. Made MR jobs write job history files on the default FS when the current context’s FS is different. Contributed by Benjamin Zhitomirsky. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618269 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 20:17:45 +00:00
Jason Darrell Lowe
857d134fda MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616018 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-05 21:52:22 +00:00
Xuan Gong
e52f67e389 YARN-1994. Expose YARN/MR endpoints on multiple interfaces. Contributed by Craig Welch, Milan Potocnik,and Arpit Agarwal 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614981 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 20:06:02 +00:00
Jian He
875592220f MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving RM-restart. Contributed by Rohith 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611434 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 18:46:20 +00:00
Karthik Kambatla
43a12f3c01 MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611196 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-16 21:24:51 +00:00
Zhijie Shen
64306aa1b5 MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current attempt is the last retry. Contributed by Wangda Tan. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609649 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-11 08:45:40 +00:00
Vinod Kumar Vavilapalli
4b2ded8202 MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side resource configuration for deciding uber-mode on map-only jobs. Contributed by Siqi Li. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1608595 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-07 20:14:01 +00:00
Zhijie Shen
c6a09d2110 MAPREDUCE-5900. Changed to the interpret container preemption exit code as a task attempt killing event. Contributed by Mayank Bansal. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607512 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-03 01:43:56 +00:00
Vinod Kumar Vavilapalli
e285b98f0f YARN-2152. Added missing information into ContainerTokenIdentifier so that NodeManagers can report the same to RM when RM restarts. Contributed Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605205 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 21:43:22 +00:00
Karthik Kambatla
7b9c074b76 MAPREDUCE-5844. Add a configurable delay to reducer-preemption. (Maysam Yabandeh via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603957 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-19 17:22:56 +00:00
Jian He
0abddac476 MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event at COMMIT_PENDING state. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603338 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-18 00:51:39 +00:00
Andrew Wang
4a839f0cc1 Fix CRLFs in TestKillAMPreemptionPolicy.java 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1596058 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-19 21:18:55 +00:00
Devarajulu K
f74e446355 MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to ProportionalCapacityPreemptionPolicy. Contributed by Sunil G. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1595754 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-19 06:42:29 +00:00
Jason Darrell Lowe
bb7ce82816 MAPREDUCE-5812. Make job context available to OutputCommitter.isRecoverySupported(). Contributed by Mohammad Kamrul Islam 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1590668 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-28 15:18:02 +00:00
Jason Darrell Lowe
283a3a246a MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state. Contributed by Ming Ma 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1590168 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-25 22:26:24 +00:00
Jason Darrell Lowe
5492149c3c MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed. Contributed by Jason Lowe 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1590046 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-25 14:53:17 +00:00
Jason Darrell Lowe
6957745c2c MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill. Contributed by Sangjin Lee 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1589524 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-23 21:53:06 +00:00