big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
10,690 Commits 2 Branches 2 Tags 264 MiB
f8204e241d
Commit Graph

693 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli
14e2639fd0 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 2014-09-13 18:04:05 -07:00
XuanGong
e65ae575a0 YARN-2456. Possible livelock in CapacityScheduler when RM is recovering 
apps. Contributed by Jian He
 2014-09-12 15:21:46 -07:00
Jian He
3122daa802 YARN-2229. Changed the integer field of ContainerId to be long type. Contributed by Tsuyoshi OZAWA 2014-09-12 10:33:33 -07:00
junping_du
6b8b1608e6 YARN-2033. Merging generic-history into the Timeline Store (Contributed by Zhijie Shen) 2014-09-12 10:04:51 +08:00
Karthik Kambatla
c11ada5ea6 YARN-2534. FairScheduler: Potential integer overflow calculating totalMaxShare. (Zhihai Xu via kasha) 2014-09-11 12:06:06 -07:00
Jian He
83be3ad444 YARN-415. Capture aggregate memory allocation at the app-level for chargeback. Contributed by Eric Payne & Andrey Klochkov 2014-09-10 18:20:54 -07:00
Jian He
cbfe26370b YARN-2158. Fixed TestRMWebServicesAppsModification#testSingleAppKill test failure. Contributed by Varun Vasudev 2014-09-10 12:47:34 -07:00
XUAN
47bdfa044a YARN-2459. RM crashes if App gets rejected for any reason and HA is enabled. Contributed by Jian He 2014-09-10 11:44:41 -07:00
Vinod Kumar Vavilapalli
b67d5ba784 YARN-2448. Changed ApplicationMasterProtocol to expose RM-recognized resource types to the AMs. Contributed by Varun Vasudev. 2014-09-10 10:15:47 -07:00
Karthik Kambatla
3072c83b38 YARN-1458. FairScheduler: Zero weight can lead to livelock. (Zhihai Xu via kasha) 2014-09-10 08:26:14 -07:00
Karthik Kambatla
1dcaba9a7a YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) 2014-09-03 10:27:36 -07:00
Karthik Kambatla
0f34e6f387 YARN-2395. FairScheduler: Preemption timeout should be configurable per queue. (Wei Yan via kasha) 2014-08-30 01:17:13 -07:00
Jian He
5c14bc426b YARN-1506. Changed RMNode/SchedulerNode to update resource with event notification. Contributed by Junping Du 2014-08-29 23:05:51 -07:00
Jian He
c686aa3533 YARN-2447. RM web service app submission doesn't pass secrets correctly. Contributed by Varun Vasudev 2014-08-29 11:40:47 -07:00
Karthik Kambatla
fa80ca49bd YARN-2405. NPE in FairSchedulerAppsBlock. (Tsuyoshi Ozawa via kasha) 2014-08-28 23:21:37 -07:00
Karthik Kambatla
d16bfd1d0f YARN-1326. RM should log using RMStore at startup time. (Tsuyoshi Ozawa via kasha) 2014-08-27 01:43:58 -07:00
Karthik Kambatla
0097b15e21 YARN-2393. FairScheduler: Add the notion of steady fair share. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619845 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 15:44:47 +00:00
Jason Darrell Lowe
4236c6600e YARN-2434. RM should not recover containers from previously failed attempt when AM restart is not enabled. Contributed by Jian He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619614 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 22:41:34 +00:00
Zhijie Shen
f6a778c372 YARN-2249. Avoided AM release requests being lost on work preserving RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618972 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-19 20:33:49 +00:00
Jian He
375c221960 YARN-2409. RM ActiveToStandBy transition missing stoping previous rmDispatcher. Contributed by Rohith 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618915 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-19 17:49:39 +00:00
Jian He
519c4be95a YARN-2411. Support simple user and group mappings to queues. Contributed by Ram Venkatesh 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618542 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 06:08:45 +00:00
Jian He
c3084d6c16 YARN-2389. Added functionality for schedulers to kill all applications in a queue. Contributed by Subramaniam Venkatraman Krishnan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618294 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 23:53:57 +00:00
Jian He
7360cec692 YARN-2378. Added support for moving applications across queues in CapacityScheduler. Contributed by Subramaniam Venkatraman Krishnan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618106 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 06:00:31 +00:00
Zhijie Shen
a9023c2736 YARN-2397. Avoided loading two authentication filters for RM and TS web interfaces. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618054 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-14 21:17:20 +00:00
Karthik Kambatla
5197f8c3c5 YARN-1370. Fair scheduler to re-populate container allocation state. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617645 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-13 01:38:59 +00:00
Karthik Kambatla
4239695588 YARN-2399. Delete old versions of files. FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617619 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 22:51:57 +00:00
Karthik Kambatla
486e718fc1 YARN-2399. FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617600 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 21:43:27 +00:00
Junping Du
c2febdcbaa YARN-1337. Recover containers upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617448 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 10:56:13 +00:00
Jian He
c4dc685343 YARN-2138. Cleaned up notifyDone* APIs in RMStateStore. Contributed by Varun Saxena 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617341 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 18:24:24 +00:00
Xuan Gong
946be75704 YARN-2400: Addendum fix for TestAMRestart failure. Contributed by Jian He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617333 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 17:42:53 +00:00
Xuan Gong
743f7f30da YARN-2400. Fixed TestAMRestart fails intermittently. Contributed by Jian He: 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617028 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-09 23:31:11 +00:00
Karthik Kambatla
a7643f4de7 YARN-2026. Fair scheduler: Consider only active queues for computing fairshare. (Ashwin Shankar via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616915 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-09 02:10:00 +00:00
Xuan Gong
eeb4acd955 YARN-2212: ApplicationMaster needs to find a way to update the AMRMToken periodically. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616892 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 21:38:24 +00:00
Karthik Kambatla
14864e9c7c YARN-2352. FairScheduler: Collect metrics on duration of critical methods that affect performance. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616769 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 14:17:54 +00:00
Jian He
8437df8ba9 YARN-2008. Fixed CapacityScheduler to calculate headroom based on max available capacity instead of configured max capacity. Contributed by Craig Welch 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616580 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 20:00:04 +00:00
Karthik Kambatla
8feddc4c84 YARN-2359. Application hangs when it fails to launch AM container. (Zhihai Xu via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616375 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 00:06:17 +00:00
Junping Du
b8f151231b YARN-1354. Recover applications upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1615550 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-04 13:25:37 +00:00
Xuan Gong
e52f67e389 YARN-1994. Expose YARN/MR endpoints on multiple interfaces. Contributed by Craig Welch, Milan Potocnik,and Arpit Agarwal 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614981 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 20:06:02 +00:00
Zhijie Shen
1d6e178144 YARN-2347. Consolidated RMStateVersion and NMDBSchemaVersion into Version in yarn-server-common. Contributed by Junping Du. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614838 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 09:27:43 +00:00
Karthik Kambatla
c0b49ff107 YARN-2328. FairScheduler: Verify update and continuous scheduling threads are stopped when the scheduler is stopped. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614432 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-29 17:41:52 +00:00
Zhijie Shen
d6532d3a77 YARN-2247. Made RM web services authenticate users via kerberos and delegation token. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613821 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-27 17:55:06 +00:00
Jian He
d4fec34933 YARN-2211. Persist AMRMToken master key in RMStateStore for RM recovery. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613515 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-25 20:42:37 +00:00
Karthik Kambatla
1e553858f9 YARN-2214. FairScheduler: preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness. (Ashwin Shankar via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613459 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-25 16:13:07 +00:00
Jason Darrell Lowe
28fca92521 YARN-2147. client lacks delegation token exception details when application submit fails. Contributed by Chen He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612950 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-23 21:40:57 +00:00
Sanford Ryza
c88402f36d YARN-2313. Livelock can occur in FairScheduler when there are lots of running apps (Tsuyoshi Ozawa via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612769 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-23 05:00:52 +00:00
Karthik Kambatla
ff77582991 YARN-2273. NPE in ContinuousScheduling thread when we lose a node. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612720 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 22:44:38 +00:00
Zhijie Shen
eac0701c96 YARN-2319. Made the MiniKdc instance start/close before/after the class of TestRMWebServicesDelegationTokens. Contributed by Wenwu Peng. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612588 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 15:15:29 +00:00
Junping Du
afb9394c91 YARN-2242. Addendum patch. Improve exception information on AM launch crashes. (Contributed by Li Lu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612565 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 13:07:23 +00:00
Karthik Kambatla
8871d8ed9f YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611840 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-19 00:12:05 +00:00
Xuan Gong
f1b831ccfb YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611820 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-18 21:46:29 +00:00
Jian He
3c193811ca YARN-2219. Addendum patch for YARN-2219 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611240 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 03:28:39 +00:00
Vinod Kumar Vavilapalli
bda23181bf YARN-2219. Changed ResourceManager to avoid AMs and NMs getting exceptions after RM recovery but before scheduler learns about apps and app-attempts. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611222 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 00:14:56 +00:00
Vinod Kumar Vavilapalli
030580387a YARN-2233. Implemented ResourceManager web-services to create, renew and cancel delegation tokens. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610876 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-15 23:00:17 +00:00
Mayank Bansal
43589a8df7 YARN-1408 Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task timeout for 30mins. (Sunil G via mayank) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610860 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-15 21:48:58 +00:00
Vinod Kumar Vavilapalli
c6cc6a6a8e YARN-2260. Fixed ResourceManager's RMNode to correctly remember containers when nodes resync during work-preserving RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610557 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-14 23:32:03 +00:00
Jian He
c9fb040c87 YARN-2181. Added preemption info to logs and RM web UI. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609561 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-10 20:03:35 +00:00
Karthik Kambatla
8fbca62a90 YARN-2131. Add a way to format the RMStateStore. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609278 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-09 19:58:43 +00:00
Zhijie Shen
12c4197b35 YARN-2158. Improved assertion messages of TestRMWebServicesAppsModification. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1608667 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-08 05:50:04 +00:00
Sanford Ryza
5644f529f3 YARN-2250. FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical (Krisztian Horvath via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607872 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-04 15:16:43 +00:00
Junping Du
5cb489f9d3 YARN-2242. Improve exception information on AM launch crashes. (Contributed by Li Lu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607655 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-03 14:15:19 +00:00
Vinod Kumar Vavilapalli
45b191e38c YARN-2232. Fixed ResourceManager to allow DelegationToken owners to be able to cancel their own tokens in secure mode. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607484 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 21:36:42 +00:00
Mayank Bansal
03a25d2cc1 YARN-2022 Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy (Sunil G via mayank) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607227 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 01:54:47 +00:00
Vinod Kumar Vavilapalli
075ff276ca YARN-1713. Added get-new-app and submit-app functionality to RM web services. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607216 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 00:23:07 +00:00
Xuan Gong
e5ae7c55d1 TestRMApplicationHistoryWriter sometimes fails in trunk. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606835 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-30 16:51:22 +00:00
Jian He
b0c51504c4 YARN-2052. Embedded an epoch number in container id to ensure the uniqueness of container id after RM restarts. Contributed by Tsuyoshi OZAWA 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606557 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-29 18:24:03 +00:00
Jian He
b717d44b52 YARN-614. Changed ResourceManager to not count disk failure, node loss and RM restart towards app failures. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606407 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-28 23:37:46 +00:00
Zhijie Shen
55a0aa0bad YARN-2201. Made TestRMWebServicesAppsModification be independent of the changes on yarn-default.xml. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606285 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-28 03:30:44 +00:00
Karthik Kambatla
f911f5495b YARN-2204. Addendum patch. TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606168 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-27 18:09:41 +00:00
Vinod Kumar Vavilapalli
9571db19eb YARN-2171. Improved CapacityScheduling to not lock on nodemanager-count when AMs heartbeat in. Contributed by Jason Lowe. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605616 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 21:56:42 +00:00
Karthik Kambatla
1a3a7e0c1a YARN-2204. TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605548 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 18:50:53 +00:00
Jian He
c3f1c30e65 YARN-1365. Changed ApplicationMasterService to allow an app to re-register after RM restart. Contributed by Anubhav Dhoot 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605263 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 04:42:39 +00:00
Vinod Kumar Vavilapalli
e285b98f0f YARN-2152. Added missing information into ContainerTokenIdentifier so that NodeManagers can report the same to RM when RM restarts. Contributed Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605205 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 21:43:22 +00:00
Thomas Graves
1f9a0fd927 YARN-2072. RM/NM UIs and webservices are missing vcore information. (Nathan Roberts via tgraves) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605162 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 19:34:34 +00:00
Karthik Kambatla
c0991d11eb YARN-2109. Fix TestRM to work with both schedulers. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605142 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 17:30:53 +00:00
Karthik Kambatla
db4d277117 YARN-2192. TestRMHA fails when run with a mix of Schedulers. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605138 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 17:05:35 +00:00
Sanford Ryza
29c102cad0 YARN-2111. In FairScheduler.attemptScheduling, we don't count containers as assigned if they have 0 memory but non-zero cores (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605113 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 15:40:39 +00:00
Vinod Kumar Vavilapalli
d16470025a YARN-2074. Changed ResourceManager to not count AM preemptions towards app failures. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605106 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 15:15:12 +00:00
Jian He
59b5e9fa15 YARN-2191. Added a new test to ensure NM will clean up completed applications in the case of RM restart. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1604949 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-23 22:52:38 +00:00
Karthik Kambatla
6fcbf9b848 YARN-2187. FairScheduler: Disable max-AM-share check by default. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1604321 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-21 07:30:07 +00:00
Jian He
95897ca14b YARN-1885. Fixed a bug that RM may not send application-clean-up signal to NMs where the completed applications previously ran in case of RM restart. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603028 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-16 23:56:12 +00:00
Vinod Kumar Vavilapalli
dc7dd1fa19 YARN-1702. Added kill app functionality to RM web services. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1602298 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-12 21:31:52 +00:00
Karthik Kambatla
4bc91b44c9 YARN-2155. FairScheduler: Incorrect threshold check for preemption. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1602295 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-12 21:23:32 +00:00
Jian He
710a8693e5 YARN-2124. Fixed NPE in ProportionalCapacityPreemptionPolicy. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601964 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-11 17:30:18 +00:00
Karthik Kambatla
5de6f72054 YARN-1424. RMAppAttemptImpl should return the DummyApplicationResourceUsageReport for all invalid accesses. (Ray Chiang via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601742 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-10 19:03:06 +00:00
Jian He
c94f2cec3a Augmented RMStateStore with state machine. Contributed by Binglin Chang. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601491 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-09 19:44:31 +00:00
Vinod Kumar Vavilapalli
424fd9494f YARN-1368. Added core functionality of recovering container state into schedulers after ResourceManager Restart so as to preserve running work in the cluster. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601303 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-09 03:09:21 +00:00
Karthik Kambatla
85d4c787e0 YARN-2128. FairScheduler: Incorrect calculation of amResource usage. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601050 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-07 01:21:33 +00:00
Junping Du
0ceb742549 YARN-1977. Add tests on getApplicationRequest with filtering start time range. (Contributed by Junping Du) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600644 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-05 13:15:44 +00:00
Sanford Ryza
16caa3fd18 YARN-1913. With Fair Scheduler, cluster can logjam when all resources are consumed by AMs (Wei Yan via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1599400 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-03 00:56:48 +00:00
Karthik Kambatla
0aad2d56df YARN-1550. NPE in FairSchedulerAppsBlock#render. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1599345 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-02 20:22:52 +00:00
Karthik Kambatla
a4ba451802 YARN-1474. Make schedulers services. (Tsuyoshi Ozawa via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598908 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-31 19:33:09 +00:00
Vinod Kumar Vavilapalli
23c325ad47 YARN-2115. Replaced RegisterNodeManagerRequest's ContainerStatus with a new NMContainerStatus which has more information that is needed for work-preserving RM-restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598790 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-31 00:20:50 +00:00
Karthik Kambatla
49a3a0cd0c YARN-2054. Better defaults for YARN ZK configs for retries and retry-inteval when HA is enabled. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598630 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-30 15:24:49 +00:00
Arpit Agarwal
4a4868e523 HADOOP-10448. Support pluggable mechanism to specify proxy user settings (Contributed by Benoy Antony) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598396 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-29 20:52:01 +00:00
Sanford Ryza
342da5b4d3 YARN-596. Use scheduling policies throughout the queue hierarchy to decide which containers to preempt (Wei Yan via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598197 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-29 04:01:24 +00:00
Sanford Ryza
edfbc8ad4a YARN-2105. Fix TestFairScheduler after YARN-2012. (Ashwin Shankar via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1597902 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-27 23:46:22 +00:00
Karthik Kambatla
7dd378c274 YARN-2096. Race in TestRMRestart#testQueueMetricsOnRMRestart. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1597223 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-23 23:51:00 +00:00
Sanford Ryza
a00b2d4f37 YARN-2073. Fair Scheduler: Add a utilization threshold to prevent preempting resources when cluster is free (Karthik Kambatla via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1597209 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-23 22:52:46 +00:00
Sanford Ryza
6c56612af5 YARN-2012. Fair Scheduler: allow default queue placement rule to take an arbitrary queue (Ashwin Shankar via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1597204 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-23 22:38:52 +00:00
Vinod Kumar Vavilapalli
82f3454f5a YARN-2017. Merged some of the common scheduler code. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1596753 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-22 05:32:26 +00:00
Jian He
0f9147c857 YARN-2053. Fixed a bug in AMS to not add null NMToken into NMTokens list from previous attempts for work-preserving AM restart. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1595116 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-16 06:22:22 +00:00
Sanford Ryza
84dfae2f8a YARN-1986. In Fifo Scheduler, node heartbeat in between creating app and attempt causes NPE (Hong Zhiguo via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1594476 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-14 06:41:20 +00:00
Christopher Douglas
45b42676f9 YARN-1957. Consider the max capacity of the queue when computing the ideal 
capacity for preemption. Contributed by Carlo Curino


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1594414 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-13 23:15:27 +00:00
Jonathan Turner Eagles
1c48142807 YARN-1981. Nodemanager version is not updated when a node reconnects (Jason Lowe via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1594358 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-13 20:03:58 +00:00
Jian He
41344a4a69 YARN-1975. Fix yarn application CLI to print the scheme of the tracking url of failed/killed applications. Contributed by Junping Du 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1593874 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-12 00:43:35 +00:00
Junping Du
ca95af7d23 YARN-2011. Fix typo and warning in TestLeafQueue (Contributed by Chen He) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1593804 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-11 15:13:29 +00:00
Sanford Ryza
cfc97a4e88 YARN-1864. Fair Scheduler Dynamic Hierarchical User Queues (Ashwin Shankar via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1593190 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-08 07:21:11 +00:00
Arpit Agarwal
f4b687b873 YARN-2018. TestClientRMService.testTokenRenewalWrongUser fails after HADOOP-10562. (Contributed by Ming Ma) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1592783 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-06 15:45:49 +00:00
Junping Du
2ad1cee5da YARN-1201. TestAMAuthorization fails with local hostname cannot be resolved. (Wangda Tan via junping_du) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1592197 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-03 13:03:27 +00:00
Vinod Kumar Vavilapalli
7a241aee90 YARN-1929. Fixed a deadlock in ResourceManager that occurs when failover happens right at the time of shutdown. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1591071 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-29 19:49:44 +00:00
Jason Darrell Lowe
a9775b4e49 YARN-738. TestClientRMTokens is failing irregularly while running all yarn tests. Contributed by Ming Ma 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1591030 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-29 17:47:11 +00:00
Chris Nauroth
84388525a3 YARN-1970. Prepare YARN codebase for JUnit 4.11. Contributed by Chris Nauroth. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1589001 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-21 23:31:18 +00:00
Vinod Kumar Vavilapalli
bad021534c YARN-1281. Fixed TestZKRMStateStoreZKClientConnections to not fail intermittently due to ZK-client timeouts. Contributed by Tsuyoshi Ozawa. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1588369 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-17 20:57:15 +00:00
Junping Du
bd43d2481e YARN-1947. TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing intermittently. (Jian He via junping_du) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1588365 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-17 20:27:37 +00:00
Vinod Kumar Vavilapalli
eb7b33c298 YARN-1928. Fixed a race condition in TestAMRMRPCNodeUpdates which caused it to fail occassionally. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1587114 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-13 22:40:16 +00:00
Vinod Kumar Vavilapalli
c6b70f4760 YARN-1933. Fixed test issues with TestAMRestart and TestNodeHealthService. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1587104 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-13 21:51:38 +00:00
Zhijie Shen
dc84800184 YARN-1924. Made ZKRMStateStore updateApplication(Attempt)StateInternal work when Application(Attempt) state hasn't been stored before. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1586547 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-11 03:36:36 +00:00
Vinod Kumar Vavilapalli
791f2f6f1b YARN-1910. Fixed a race condition in TestAMRMTokens that causes the test to fail more often on Windows. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1586192 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-10 01:36:50 +00:00
Kihwal Lee
c3f6d71726 YARN-1907. TestRMApplicationHistoryWriter#testRMWritingMassiveHistory intermittently fails. Contributed by Mit Desai. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1585992 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-09 15:33:25 +00:00
Karthik Kambatla
1e1200d594 YARN-1784. TestContainerAllocation assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1585894 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-09 05:49:24 +00:00
Jian He
15ac36a216 YARN-1837. Fixed TestMoveApplication#testMoveRejectedByScheduler failure. Contributed by Hong Zhiguo 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1584862 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-04 18:42:35 +00:00
Jonathan Turner Eagles
6d2e1fde63 YARN-1883. TestRMAdminService fails due to inconsistent entries in UserGroups (Mit Desai via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1582862 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-28 19:30:08 +00:00
Tsz-wo Sze
3f7aa79e58 HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure, getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from DistributedFileSystem. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1582856 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-28 18:29:31 +00:00
Jian He
71c2b159ab YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1581678 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-26 03:43:41 +00:00
Vinod Kumar Vavilapalli
8a9ae9e3ec YARN-1867. Fixed a bug in ResourceManager that was causing invalid ACL checks in the web-services after fail-over. Contributed by Vinod Kumar Vavilapalli. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1581662 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-26 03:03:17 +00:00
Vinod Kumar Vavilapalli
d14eff7d38 YARN-1866. Fixed an issue with renewal of RM-delegation tokens on restart or fail-over. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1581448 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-25 18:21:55 +00:00
Jian He
0febaa1b3d YARN-1852. Fixed RMAppAttempt to not resend AttemptFailed/AttemptKilled events to already recovered Failed/Killed RMApps. Contributed by Rohith Sharmaks 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580997 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-24 19:04:57 +00:00
Jian He
ce56616037 YARN-1776. Fixed DelegationToken renewal to survive RM failover. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580154 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-22 04:12:57 +00:00
Karthik Kambatla
cbb3914c89 YARN-1536. Cleanup: Get rid of ResourceManager#get*SecretManager() methods and use the RMContext methods instead. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580128 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-22 02:35:52 +00:00
Vinod Kumar Vavilapalli
dc594101cc YARN-1854. Fixed test failure in TestRMHA#testStartAndTransitions. Contributed by Rohith Sharma KS. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580097 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-22 00:29:43 +00:00
Jian He
f67218809c YARN-1849. Fixed NPE in ResourceTrackerService#registerNodeManager for UAM. Contributed by Karthik Kambatla 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1580077 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-21 22:53:49 +00:00
Karthik Kambatla
bfafc1d0ce YARN-1705. Reset cluster-metrics on transition to standby. (Rohith via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1579014 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-18 19:00:46 +00:00
Karthik Kambatla
e52c1535b8 YARN-1846. TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578763 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-18 06:56:15 +00:00
Vinod Kumar Vavilapalli
57cdf8626a YARN-1512. Enhanced CapacityScheduler to be able to decouple scheduling from node-heartbeats. Contributed by Arun C Murthy. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578722 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-18 02:53:20 +00:00
Vinod Kumar Vavilapalli
92317e3459 YARN-1839. Fixed handling of NMTokens in ResourceManager such that containers launched by AMs running on the same machine as the AM are correctly propagated. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578631 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 22:22:15 +00:00
Vinod Kumar Vavilapalli
81a456e638 YARN-1591. Fixed AsyncDispatcher to handle interrupts on shutdown in a sane manner and thus fix failure of TestResourceTrackerService. Contributed by Tsuyoshi Ozawa. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578628 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 22:17:43 +00:00
Vinod Kumar Vavilapalli
8314674947 YARN-1685. Fixed few bugs related to handling of containers' log-URLs on ResourceManager and history-service. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578602 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 21:36:21 +00:00
Jonathan Turner Eagles
7944eab64a YARN-1136. Replace junit.framework.Assert with org.junit.Assert (Chen He via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578539 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 20:03:35 +00:00
Jian He
bc00bc966b YARN-1830. Fixed TestRMRestart#testQueueMetricsOnRMRestart failure due to race condition when app is submitted. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1578486 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-17 18:03:06 +00:00
Jonathan Turner Eagles
95b7f50813 YARN-1833. TestRMAdminService Fails in trunk and branch-2 (Mit Desais via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1577737 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-14 21:58:42 +00:00
Zhijie Shen
6af044bb75 YARN-1389. Made ApplicationClientProtocol and ApplicationHistoryProtocol expose analogous getApplication(s)/Attempt(s)/Container(s) APIs. Contributed by Mayank Bansal. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1577052 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-13 07:42:29 +00:00
Jason Darrell Lowe
135700a611 YARN-1789. ApplicationSummary does not escape newlines in the app name. Contributed by Tsuyoshi OZAWA 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576960 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 22:25:01 +00:00
Vinod Kumar Vavilapalli
197217c95d YARN-1816. Fixed ResourceManager to get RMApp correctly handle ATTEMPT_FINISHED event at ACCEPTED state that can happen after RM restarts. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576911 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 21:18:55 +00:00
Vinod Kumar Vavilapalli
4de17c6052 YARN-1812. Fixed ResourceManager to synchrously renew tokens after recovery and thus recover app itself synchronously and avoid races with resyncing NodeManagers. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576843 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 18:25:33 +00:00
Arun Murthy
4ce0e4bf2e YARN-1444. Fix CapacityScheduler to deal with cases where applications specify host/rack requests without off-switch request. Contributed by Wangda Tan. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576751 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 14:36:27 +00:00
Karthik Kambatla
477ed62b3f YARN-1821. NPE on registerNodeManager if the request has containers for UnmanagedAMs (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576525 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-11 22:18:54 +00:00
Vinod Kumar Vavilapalli
0edda25373 YARN-1764. Modified YarnClient to correctly handle failover of ResourceManager after the submitApplication call goes through. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576160 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-11 00:52:03 +00:00
Vinod Kumar Vavilapalli
e4190b2a97 YARN-1788. Fixed a bug in ResourceManager to set the apps-completed and apps-killed metrics correctly for killed applications. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576072 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-10 20:36:34 +00:00
Jian He
7da07461ff YARN-1793. Fixed ClientRMService#forceKillApplication not killing unmanaged application. Contributed by Karthik Kambatla 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1576023 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-10 17:58:55 +00:00
Vinod Kumar Vavilapalli
8497b870af YARN-1410. Added tests to validate that clients can fail-over to a new RM 
after getting an application-ID but before submission and can still submit
to the newly active RM with no issues. Contributed by Xuan Gong.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1575478 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-08 04:43:29 +00:00
Karthik Kambatla
b06cc16f7d YARN-1774. FS: Submitting to non-leaf queue throws NPE. (Anubhav Dhoot and Karthik Kambatla via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1575415 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-07 21:34:19 +00:00
Jian He
8cd23c5b48 YARN-1752. Fixed ApplicationMasterService to reject unregister request if AM did not register before. Contributed by Rohith Sharma. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1574623 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-05 19:04:58 +00:00
Vinod Kumar Vavilapalli
82e8c8bd71 YARN-1766. Fixed a bug in ResourceManager to use configuration loaded from the configuration-provider when booting up. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1574252 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-04 23:35:20 +00:00
Vinod Kumar Vavilapalli
88245b6a41 YARN-986. Changed client side to be able to figure out the right RM Delegation token for the right ResourceManager when HA is enabled. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1574190 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-04 20:39:06 +00:00
Vinod Kumar Vavilapalli
c26bdddcc3 YARN-1675. Added the previously missed new file. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1573736 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-03 20:39:01 +00:00
Vinod Kumar Vavilapalli
a60434368c YARN-1758. Fixed ResourceManager to not mandate the presence of site specific configuration files and thus fix failures in downstream tests. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1573695 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-03 19:41:52 +00:00
Karthik Kambatla
8caeb0af9d YARN-1760. Addendum patch. TestRMAdminService assumes Capacity Scheduler. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1573134 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-01 08:46:29 +00:00
Karthik Kambatla
1ba26e3e68 YARN-1528. Allow setting auth for ZK connections. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1573014 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-28 17:34:08 +00:00
Vinod Kumar Vavilapalli
0fbc7fe816 YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of transferred containers from previous app-attempts to new AMs after YARN-1490. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1572230 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-26 20:20:07 +00:00
Karthik Kambatla
5a42e1b7c3 YARN-1760. TestRMAdminService assumes CapacityScheduler. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1571777 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-25 18:06:45 +00:00
Vinod Kumar Vavilapalli
ad70f26b1f YARN-1734. Fixed ResourceManager to update the configurations when it transits from standby to active mode so as to assimilate any changes that happened while it was in standby mode. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1571539 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-25 02:07:19 +00:00
Vinod Kumar Vavilapalli
e167e585e9 YARN-1749. Updated application-history related configs to reflect the latest reality and to be consistently named. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1570948 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-23 02:04:58 +00:00
Zhijie Shen
da20095284 YARN-1071. Enabled ResourceManager to recover cluster metrics numDecommissionedNMs after restarting. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1570469 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-21 06:46:24 +00:00
Vinod Kumar Vavilapalli
5fd5c9900c YARN-713. Fixed ResourceManager to not crash while building tokens when DNS issues happen transmittently. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569979 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-19 23:39:13 +00:00
Sanford Ryza
0e3e91a06d YARN-1718. Fix a couple isTerminals in Fair Scheduler queue placement rules (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569928 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-19 21:36:51 +00:00
Vinod Kumar Vavilapalli
9da9f7d4d8 YARN-1666. Modified RM HA handling of include/exclude node-lists to be available across RM failover by making using of a remote configuration-provider. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569856 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-19 17:52:57 +00:00
Jonathan Turner Eagles
996acc834e YARN-1479. Invalid NaN values in Hadoop REST API JSON response (Chen He via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569853 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-19 17:49:21 +00:00
Jian He
3a7e7b3671 YARN-1428. Fixed RM to write the final state of RMApp/RMAppAttempt to the application history store in the transition to the final state. (Contributed by Zhijie Shen) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569585 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-19 01:19:07 +00:00
Vinod Kumar Vavilapalli
990cffdcfa YARN-1553. Modified YARN and MR to stop using HttpConfig.isSecure() and 
instead rely on the http policy framework. And also fix some bugs related
to https handling in YARN web-apps. Contributed by Haohui Mai.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1568501 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-14 20:01:02 +00:00
Vinod Kumar Vavilapalli
d0a5e43de7 YARN-1417. Modified RM to generate container-tokens not at creation time, but at allocation time so as to prevent RM 
from shelling out containers with expired tokens. Contributed by Omkar Vinit Joshi and Jian He.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1568060 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-13 22:02:11 +00:00
Vinod Kumar Vavilapalli
2b916db60b YARN-1676. Modified RM HA handling of user-to-group mappings to be available across RM failover by making using of a remote configuration-provider. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1568041 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-13 21:23:33 +00:00
Jian He
dab635980d YARN-1345. Remove FINAL_SAVING state from YarnApplicationAttemptState. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1567820 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-13 00:59:19 +00:00
Karthik Kambatla
41ec3cce76 YARN-1641. ZK store should attempt a write periodically to ensure it is still Active. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1567628 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-12 14:09:13 +00:00
Vinod Kumar Vavilapalli
1fa6ab249b YARN-1459. Changed ResourceManager to depend its service initialization on the configuration-provider mechanism during startup too. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1566791 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-10 22:50:15 +00:00
Vinod Kumar Vavilapalli
8b2336fcef YARN-1660. Simplified the RM HA configuration to accept and be able to simply depend just on 
configuration properties of the form yarn.resourcemanager.hostname.RMID and use the default
ports for all service addresses. Contributed by Xuan Gong.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1565523 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-07 03:18:39 +00:00
Vinod Kumar Vavilapalli
30294a2196 YARN-1665. Simplify the configuration of RM HA by having better default values. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1565517 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-07 02:39:32 +00:00
Zhijie Shen
943b2190d7 YARN-1689. Made RMAppAttempt get killed when RMApp is at ACCEPTED. Contributed by Vinod Kumar Vavilapalli. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1565497 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-07 00:18:46 +00:00
Sanford Ryza
e6eccf20de YARN-1499. Fair Scheduler changes for moving apps between queues (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564856 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-05 18:09:07 +00:00
Zhijie Shen
ebe0c17a95 YARN-1461. Added tags for YARN applications and changed RM to handle them. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564633 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-05 04:55:00 +00:00
Vinod Kumar Vavilapalli
b812af964d YARN-1669. Modified RM HA handling of protocol level service-ACLS to be available across RM failover by making using of a remote configuration-provider. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564549 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-04 22:46:52 +00:00
Vinod Kumar Vavilapalli
24fa232707 YARN-1667. Modified RM HA handling of super users (with proxying ability) to be available across RM failover by making using of a remote configuration-provider. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564100 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-03 22:10:56 +00:00
Vinod Kumar Vavilapalli
eff1e809f2 YARN-1668. Modified RM HA handling of admin-acls to be available across RM failover by making using of a remote configuration-provider. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564043 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-03 20:16:28 +00:00
Vinod Kumar Vavilapalli
3e7d56678c YARN-1639. Modified RM HA configuration handling to have a way of not requiring separate configuration files for each RM. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564032 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-03 19:53:46 +00:00
Vinod Kumar Vavilapalli
ca72e11158 YARN-1611. Introduced the concept of a configuration provider which can be used by ResourceManager to read configuration locally or from remote systems so as to help RM failover. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1564002 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-03 19:07:55 +00:00
Sanford Ryza
1079c37128 YARN-1504. RM changes for moving apps between queues (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1563371 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-01 04:03:36 +00:00
Jonathan Turner Eagles
5d4bd6a15a YARN-1632. TestApplicationMasterServices should be under org.apache.hadoop.yarn.server.resourcemanager package (Chen He via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1563157 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-31 16:00:39 +00:00
Sanford Ryza
981679e6d7 YARN-1498. Common scheduler changes for moving apps between queues (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1563021 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-31 00:14:47 +00:00
Karthik Kambatla
f1cef4cd29 YARN-1618. Fix invalid RMApp transition from NEW to FINAL_SAVING (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1562529 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-29 18:02:40 +00:00
Sanford Ryza
7f2b01a742 YARN-1629. IndexOutOfBoundsException in MaxRunningAppsEnforcer (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1561996 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-28 08:34:55 +00:00
Vinod Kumar Vavilapalli
cbee889711 YARN-321. Merging YARN-321 branch to trunk. 
svn merge ../branches/YARN-321


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1561452 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-26 04:51:10 +00:00
Sanford Ryza
163348701c YARN-1623. Include queue name in RegisterApplicationMasterResponse (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1560545 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-22 22:25:03 +00:00
Sanford Ryza
8ceddeb0c3 YARN-1607. TestRM relies on the scheduler assigning multiple containers in a single node update (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1560533 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-22 22:08:04 +00:00
Sanford Ryza
1a7e158844 YARN-1624. QueuePlacementPolicy format is not easily readable via a JAXB parser (Aditya Acharya via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1560470 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-22 18:51:23 +00:00
Junping Du
4d413350d7 YARN-1603. Remove two *.orig files which were unexpectedly committed. (Zhijie Shen via junping_du) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1558340 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-15 11:05:15 +00:00
Sanford Ryza
f4fd050dee YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and parent on allocation file reload (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1558228 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-14 22:33:56 +00:00
Vinod Kumar Vavilapalli
25bc68d15e YARN-1041. Added the ApplicationMasterProtocol API for applications to use the ability in ResourceManager to optionally not kill containers when the ApplicationMaster exits. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1557318 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-11 07:07:17 +00:00
Jian He
f677175f35 YARN-1166. Fixed app-specific and attempt-specific QueueMetrics to be triggered by accordingly app event and attempt event. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1557296 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-11 01:15:49 +00:00
Karthik Kambatla
6608b75829 YARN-1574. RMDispatcher should be reset on transition to standby. (Xuan Gong via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1557248 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-10 20:15:00 +00:00
Karthik Kambatla
a0ecb98a77 YARN-1033. Expose RM active/standby state to Web UI and REST API (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1557181 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-10 16:35:52 +00:00
Vinod Kumar Vavilapalli
1393581bce YARN-1490. Introduced the ability to make ResourceManager optionally not kill all containers when an ApplicationMaster exits. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1557143 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-10 15:04:39 +00:00
Vinod Kumar Vavilapalli
c3cc855d27 YARN-1029. Added embedded leader election in the ResourceManager. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1556103 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-07 01:56:11 +00:00
Vinod Kumar Vavilapalli
a45017a5f1 YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize app-attempts separately from apps. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1554896 13f79535-47bb-0310-9956-ffa450edef68
 2014-01-02 20:19:45 +00:00
Vinod Kumar Vavilapalli
50480f892a YARN-1522. Fixed a race condition in the test TestApplicationCleanup that was causing it to randomly fail. Contributed by Liyin Liang. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1554328 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-30 22:21:55 +00:00
Vinod Kumar Vavilapalli
8f0bf54d34 YARN-1541. Changed ResourceManager to invalidate ApplicationMaster host/port information once an AM crashes. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553772 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-28 01:09:07 +00:00
Karthik Kambatla
d841a0f474 YARN-1172. Convert SecretManagers in RM to services (Tsuyoshi OZAWA via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1553431 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-26 00:21:56 +00:00
Vinod Kumar Vavilapalli
93907baa0b YARN-1307. Redesign znode structure for Zookeeper based RM state-store for better organization and scalability. Contributed by Tsuyoshi OZAWA. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1552209 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-19 02:33:05 +00:00
Vinod Kumar Vavilapalli
b774d7b3de YARN-1446. Changed client API to retry killing application till RM acknowledges so as to account for RM crashes/failover. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1551444 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-17 02:16:20 +00:00
Karthik Kambatla
a6754bbb81 YARN-1451. TestResourceManager relies on the scheduler assigning multiple containers in a single node update. (Sandy Ryza via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1551440 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-17 01:47:23 +00:00
Vinod Kumar Vavilapalli
1182ca04d4 YARN-1485. Modified RM HA configuration validation to also ensure that service-address configuration are configured for every RM. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550854 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-13 23:54:44 +00:00
Jian He
28db28d491 YARN-1311. Fixed app specific scheduler-events' names to be app-attempt based. Contributed by Vinod Kumar Vavilapalli 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550613 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-13 01:44:16 +00:00
Jian He
791e43c53b Reverting YARN-1311. Fixed app specific scheduler-events' names to be app-attempt based. Contributed by Vinod Kumar Vavilapalli 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550594 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-12 23:45:01 +00:00
Jian He
38c32ed239 YARN-1311. Fixed app specific scheduler-events' names to be app-attempt based. Contributed by Vinod Kumar Vavilapalli 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550579 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-12 23:07:17 +00:00
Vinod Kumar Vavilapalli
8a79be36bd YARN-1325. Modified RM HA configuration validation to also ensure that multiple RMs are configured. Contributed by Xuan Gong. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550524 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-12 20:50:11 +00:00
Vinod Kumar Vavilapalli
4ff1a8203b YARN-408. Change CapacityScheduler to not disable delay-scheduling by default. Contributed by Mayank Bansal. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1550245 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-11 20:14:30 +00:00
Vinod Kumar Vavilapalli
48fb53bc49 YARN-1405. Fixed ResourceManager to not hang when init/start fails with an exception w.r.t state-store. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548992 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-08 04:30:32 +00:00
Vinod Kumar Vavilapalli
305ae48136 YARN-1378. Implemented a cleaner of old finished applications from the RM state-store. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548990 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-08 04:17:51 +00:00
Sanford Ryza
e1d3670f4c YARN-807. When querying apps by queue, iterating over all apps is inefficient and limiting (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548983 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-08 03:09:34 +00:00
Sanford Ryza
eccd7b2093 YARN-546. Allow disabling the Fair Scheduler event log. (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548360 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-06 01:32:27 +00:00
Sanford Ryza
6d5f8ebed6 YARN-1403. Separate out configuration loading from QueueManager in the Fair Scheduler (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1548006 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-05 03:26:11 +00:00
Vinod Kumar Vavilapalli
7fe953ed90 YARN-1454. Fixed test failure issue with TestRMRestart. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1547651 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-04 00:22:12 +00:00
Vinod Kumar Vavilapalli
9c95015bb4 YARN-895. Changed RM state-store to not crash immediately if RM restarts while the state-store is down. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1547538 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-03 19:28:02 +00:00
Vinod Kumar Vavilapalli
c58ae266e9 YARN-1318. Promoted AdminService to an Always-On service and merged it into RMHAProtocolService. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1547212 13f79535-47bb-0310-9956-ffa450edef68
 2013-12-02 22:18:44 +00:00
Sanford Ryza
61b6ed73f8 YARN-1241: Include missing files 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1546625 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-29 19:08:48 +00:00
Sanford Ryza
7545d8bf99 YARN-1241. In Fair Scheduler, maxRunningApps does not work for non-leaf queues. (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1546623 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-29 19:02:41 +00:00
Vinod Kumar Vavilapalli
6369c8d819 YARN-1239. Modified ResourceManager state-store implementations to start storing version numbers. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1546229 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-27 23:22:33 +00:00
Vinod Kumar Vavilapalli
6c243fceac YARN-1416. Fixed a few invalid transitions in RMApp, RMAppAttempt and in some tests. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1545448 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-25 23:10:34 +00:00
Sanford Ryza
38f94dc16d YARN-1423. Support queue placement by secondary group in the Fair Scheduler (Ted Malaska via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1545157 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-25 06:04:26 +00:00
Bikas Saha
8313697752 YARN-1425. TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument (Omkar Vinit Joshi via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543952 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-20 21:16:39 +00:00
Sanford Ryza
2f6b393543 YARN-1407. RM Web UI and REST APIs should uniformly use YarnApplicationState (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543675 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-20 00:47:18 +00:00
Vinod Kumar Vavilapalli
512475e56f YARN-674. Fixed ResourceManager to renew DelegationTokens on submission asynchronously to work around potential slowness in state-store. Contributed by Omkar Vinit Joshi. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543312 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-19 05:20:58 +00:00
Vinod Kumar Vavilapalli
cfa783141f YARN-1210. Changed RM to start new app-attempts on RM restart only after ensuring that previous AM exited or after expiry time. Contributed by Omkar Vinit Joshi. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543310 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-19 05:17:20 +00:00
Vinod Kumar Vavilapalli
4341562622 YARN-709. Added tests to verify validity of delegation tokens and logging of appsummary after RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543269 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-19 00:50:34 +00:00
Jason Darrell Lowe
e77aec001f YARN-1419. TestFifoScheduler.testAppAttemptMetrics fails intermittently under jdk7. Contributed by Jonathan Eagles 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1543117 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-18 19:24:05 +00:00
Bikas Saha
f7efa0b86e YARN-1411. HA config shouldn't affect NodeManager RPC addresses (Karthik Kambatla via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1542367 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-15 19:02:30 +00:00
Sanford Ryza
6ac4ac05f7 YARN-1392: Add new files 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1542106 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-14 22:13:07 +00:00
Sanford Ryza
3858b9018e YARN-1392. Allow sophisticated app-to-queue placement policies in the Fair Scheduler (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1542105 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-14 22:12:13 +00:00
Bikas Saha
797159bbd4 YARN-1222. Make improvements in ZKRMStateStore for fencing (Karthik Kambatla via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1541995 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-14 17:53:25 +00:00
Sanford Ryza
72c6d6255a YARN-1387. RMWebServices should use ClientRMService for filtering applications (Karthik Kambatla via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1540851 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-11 21:49:43 +00:00
Vinod Kumar Vavilapalli
e28015ed1b YARN-1121. Changed ResourceManager's state-store to drain all events on shut-down. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1540232 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-09 00:55:21 +00:00
Luke Lu
6990355e57 YARN-311. RM/scheduler support for dynamic resource configuration. (Junping Du via llu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1539134 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-05 21:23:53 +00:00
Vinod Kumar Vavilapalli
4550eac099 YARN-1374. Changed ResourceManager to start the preemption policy monitors as active services. Contributed by Karthik Kambatla. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1539089 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-05 18:25:30 +00:00
Sanford Ryza
da317f2ea5 YARN-786. Expose application resource usage in RM REST API (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1537809 13f79535-47bb-0310-9956-ffa450edef68
 2013-11-01 02:54:28 +00:00
Sanford Ryza
30007fd686 YARN-1290. Let continuous scheduling achieve more balanced task assignment (Wei Yan via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1537731 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-31 22:04:31 +00:00
Vinod Kumar Vavilapalli
9f4d4e27fb YARN-891. Modified ResourceManager state-store to remember completed applications so that clients can get information about them post RM-restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1537560 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-31 17:25:06 +00:00
Alejandro Abdelnur
990dcae717 YARN-1343. NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs. (tucu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1537368 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-31 02:54:39 +00:00
Bikas Saha
03510d00f4 YARN-1068. Add admin support for HA operations (Karthik Kambatla via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1536888 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-29 21:03:54 +00:00
Sanford Ryza
025f171947 YARN-1333. Support blacklisting in the Fair Scheduler (Tsuyoshi Ozawa via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1535899 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-26 00:27:40 +00:00
Sanford Ryza
3baff29b8c YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1534861 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-22 23:53:32 +00:00
Sanford Ryza
755def8fe9 YARN-1315. TestQueueACLs should also test FairScheduler (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1534508 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-22 06:22:46 +00:00
Sanford Ryza
dc523bd182 YARN-1288. Make Fair Scheduler ACLs more user friendly (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1534315 13f79535-47bb-0310-9956-ffa450edef68
 2013-10-21 18:45:38 +00:00