big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,668 Commits 2 Branches 2 Tags 264 MiB
3a6f8b8785
Commit Graph

398 Commits

Author SHA1 Message Date
Karthik Kambatla
1a8781f1f9 MAPREDUCE-5689. MRAppMaster does not preempt reducers when scheduled maps cannot be fulfilled. (lohit via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1555161 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-03 17:27:36 +00:00
Christopher Douglas
47cca0cb6d MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state. 
Contributed by Carlo Curino


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553939 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-28 21:58:33 +00:00
Vinod Kumar Vavilapalli
cc4c74be09 MAPREDUCE-5694. Fixed MR AppMaster to shutdown the LogManager so as to avoid losing syslog in some conditions. Contributed by Mohammad Kamrul Islam. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553879 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-28 17:51:03 +00:00
Sanford Ryza
04dac63695 MAPREDUCE-5550. Task Status message (reporter.setStatus) not shown in UI with Hadoop 2.0 (Gera Shegalov via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1552808 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-20 21:00:25 +00:00
Sanford Ryza
1b9b956924 MAPREDUCE-5692. Add explicit diagnostics when a task attempt is killed due to speculative execution (Gera Shegalov via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1552797 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-20 19:53:59 +00:00
Christopher Douglas
9ca394d54d MAPREDUCE-5189. Add policies and wiring to respond to preemption requests 
from YARN. Contributed by Carlo Curino.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1551748 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-17 22:54:31 +00:00
Jonathan Turner Eagles
ee5351bf22 MAPREDUCE-5052. Job History UI and web services confusing job start time and job submit time (Chen He via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1549896 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-10 17:17:47 +00:00
Jason Darrell Lowe
0f2ce45aeb MAPREDUCE-5409. MRAppMaster throws InvalidStateTransitonException: Invalid event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl. Contributed by Gera Shegalov 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548197 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-05 16:30:12 +00:00
Jonathan Turner Eagles
9ba98e6af1 MAPREDUCE-5632. TestRMContainerAllocator#testUpdatedNodes fails (jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1547929 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-04 22:11:28 +00:00
Jing Zhao
2214871d91 HDFS-5545. Allow specifying endpoints for listeners in HttpServer. Contributed by Haohui Mai. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1546151 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-27 18:20:14 +00:00
Jason Darrell Lowe
0a5b0499f3 MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1544266 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-21 17:53:39 +00:00
Jonathan Turner Eagles
a02e60fefc MAPREDUCE-5373. TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently (jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1542456 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-16 03:29:51 +00:00
Sanford Ryza
3d95049f79 MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1541844 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-14 07:56:11 +00:00
Sanford Ryza
e94ed91114 MAPREDUCE-5613. DefaultSpeculator holds and checks hashmap that is alway empty (Gera Shegalov via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1541433 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-13 09:23:18 +00:00
Jason Darrell Lowe
298bddf00b MAPREDUCE-5561. org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl testcase failing on trunk. Contributed by Karthik Kambatla 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1534832 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-22 22:42:43 +00:00
Siddharth Seth
f2745babf3 MAPREDUCE-5329. Allow MR applications to use additional AuxServices, which are compatible with the default MapReduce shuffle. Contributed by Avner BenHanoch. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1531741 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-13 20:17:53 +00:00
Vinod Kumar Vavilapalli
f0799c5536 MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir cleanup, sending job-end notification correctly when unregister with RM fails. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1529682 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-06 20:53:28 +00:00
Vinod Kumar Vavilapalli
21181b6553 YARN-1277. Added a policy based configuration for http/https in common HttpServer and using the same in YARN - related 
to per project https config support via HADOOP-10022. Contributed by Suresh Srinivas and Omkar Vinit Joshi.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1529662 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-06 18:43:36 +00:00
Vinod Kumar Vavilapalli
d4324eef14 MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that aren't heart-beating for a while, so that we can aggressively speculate instead of waiting for task-timeout. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1529229 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-04 17:26:56 +00:00
Bikas Saha
fb3f338c65 MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting feature in RM requests (Zhijie Shen via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1529005 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-03 21:14:35 +00:00
Vinod Kumar Vavilapalli
82f4348f27 MAPREDUCE-5536. Fixed MR AM and JHS to respect mapreduce.jobhistory.webapp.https.address. Contributed by Omkar Vinit Joshi. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1528251 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-01 22:52:25 +00:00
Vinod Kumar Vavilapalli
aa12a7914f MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1527219 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-28 19:14:16 +00:00
Jason Darrell Lowe
711abc3827 MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow times out. Contributed by Robert Kanter 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1527037 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-27 19:14:07 +00:00
Arun Murthy
96c11fe602 MAPREDUCE-5514. Fix TestRMContainerAllocator. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1526735 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-26 23:02:51 +00:00
Bikas Saha
524dad1109 MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1526071 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-25 00:46:48 +00:00
Vinod Kumar Vavilapalli
78e995a37d YARN-1204. Added separate configuration properties for https for RM and NM without which servers enabled with https will also start on http ports. Contributed by Omkar Vinit Joshi. 
MAPREDUCE-5523. Added separate configuration properties for https for JHS without which even when https is enabled, it starts on http port itself. Contributed by Omkar Vinit Joshi.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1525947 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-24 17:20:18 +00:00
Vinod Kumar Vavilapalli
13420d01f2 YARN-1203. Changed YARN web-app proxy to handle http and https URLs from AM registration and finish correctly. Contributed by Omkar Vinit Joshi. 
MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config mapreduce.ssl.enabled to enable https and disabling it by default as MR AM needs
to set up its own certificates etc and not depend on clusters'. Contributed by Omkar Vinit Joshi.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1524864 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-19 23:35:58 +00:00
Sanford Ryza
c09f00771e MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again in Limits (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1524408 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-18 13:44:33 +00:00
Bikas Saha
ec010a2936 YARN-540. Race condition causing RM to potentially relaunch already unregistered AMs on RM restart (Jian He via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1523376 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-15 01:01:06 +00:00
Bikas Saha
f4951e0708 YARN-1116. Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts (Jian He via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1523146 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-13 23:51:50 +00:00
Vinod Kumar Vavilapalli
1c1ebc1553 MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else but just before ClientService to avoid race conditions during RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1521699 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-11 00:38:49 +00:00
Devarajulu K
b2af6c7024 MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE. Contributed by Nemon Lou. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1520964 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-09 06:42:42 +00:00
Jason Darrell Lowe
5540d77e2f MAPREDUCE-5475. MRClientService does not verify ACLs properly. Contributed by Jason Lowe 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1520156 13f79535-47bb-0310-9956-ffa450edef68
 2013-09-04 22:23:40 +00:00
Sanford Ryza
236b8530bd HADOOP-9918. Add addIfService to CompositeService (Karthik Kambatla via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1519129 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-31 02:09:45 +00:00
Sanford Ryza
2cc851a66e MAPREDUCE-5484. YarnChild unnecessarily loads job conf twice (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1518857 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-29 23:45:16 +00:00
Vinod Kumar Vavilapalli
5d4b684c02 MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM commands to reboot, so that client can continue to track the overall job. Contributed by xJian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1518821 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-29 21:07:57 +00:00
Jason Darrell Lowe
c660339c09 Revert MAPREDUCE-5475 and YARN-707 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1517097 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-24 01:15:37 +00:00
Jason Darrell Lowe
6f93f20515 MAPREDUCE-5475. MRClientService does not verify ACLs properly. Contributed by Jason Lowe 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1517085 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-23 22:47:37 +00:00
Vinod Kumar Vavilapalli
740f4cb97a MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory only after unregistering from the RM. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1516660 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-22 23:17:12 +00:00
Jason Darrell Lowe
d2c70e3e94 Revert MAPREDUCE-5475 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1516594 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-22 20:34:46 +00:00
Arun Murthy
148bf3ea4e MAPREDUCE-5475. Ensure MRClientService verifies ACLs for users. Contributed by Jason Lowe. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1516361 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-22 01:59:25 +00:00
Arun Murthy
ded91b4cfa MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. Contributed by Vinod K. V. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1516358 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-22 01:52:24 +00:00
Vinod Kumar Vavilapalli
2d614a916c MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate AMs in case they are exiting because of errors and thus help history-server pick up the right history file for the last successful AM. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1516238 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-21 18:16:47 +00:00
Kihwal Lee
34ae96fd8c MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race conditions. Contributed by Jason Lowe. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1510581 13f79535-47bb-0310-9956-ffa450edef68
 2013-08-05 16:03:02 +00:00
Tsz-wo Sze
a690a215db HADOOP-9756. Remove the deprecated getServer(..) methods from RPC. Contributed by Junping Du 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1507259 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-26 10:59:20 +00:00
Vinod Kumar Vavilapalli
3ca3137179 YARN-926. Modified ContainerManagerProtcol APIs to take in requests for multiple containers. Contributed by Jian He. 
MAPREDUCE-5412. Update MR app to use multiple containers API of ContainerManager after YARN-926. Contributed by Jian He.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1506391 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-24 03:41:03 +00:00
Jason Darrell Lowe
0939c49368 MAPREDUCE-5317. Stale files left behind for failed jobs. Contributed by Ravi Prakash 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1506154 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-23 16:52:24 +00:00
Arun Murthy
ac914f79bc YARN-918. Remove ApplicationAttemptId from RegisterApplicationMasterRequestProto. Contributed by Vinod K V. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1504735 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-19 00:57:40 +00:00
Arun Murthy
32bc200d54 YARN-701. Use application tokens irrespective of secure or non-secure mode. Contributed by Vinod K V. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1504604 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-18 19:03:48 +00:00
Bikas Saha
bf876f70fb MAPREDUCE-5398. MR changes for YARN-513 (Jian He via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1503934 13f79535-47bb-0310-9956-ffa450edef68
 2013-07-16 22:51:12 +00:00