big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,697 Commits 2 Branches 2 Tags 264 MiB
dbb9dded33
Commit Graph

1928 Commits

Author SHA1 Message Date
Sunil G
a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G
f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Robert Kanter
d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Robert Kanter
c83fe44917 YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 2017-12-01 12:18:13 -08:00
Wangda Tan
a63d19d365 YARN-6124. Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues. (Zian Chen via wangda) 
Change-Id: Id93656f3af7dcd78cafa94e33663c78d410d43c2
 2017-11-30 15:57:22 -08:00
Daniel Templeton
8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Jason Lowe
a2c7a73e33 YARN-6647. RM can crash during transitionToStandby due to InterruptedException. Contributed by Bibin A Chundatt 2017-11-28 11:15:44 -06:00
Jian He
fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Yufei Gu
2bde3aedf1 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 2017-11-24 23:32:46 -08:00
Wangda Tan
834e91ee91 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
 2017-11-23 19:59:03 -08:00
Arun Suresh
b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Yufei Gu
4cc9479dae YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 2017-11-22 14:18:36 -08:00
yufei
03c311eae3 YARN-7513. Remove the scheduler lock in FSAppAttempt.getWeight() (Contributed by Wilfred Spiegelenburg) 2017-11-21 10:33:34 -08:00
Wangda Tan
0d781dd03b YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
 2017-11-20 11:48:15 -08:00
bibinchundatt
b5b81a4f08 YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 2017-11-18 19:25:29 +05:30
Eric Yang
0940e4f692 YARN-7218. Decouple YARN Services REST API namespace from RM. (Contributed by Eric Yang) 2017-11-17 12:28:12 -05:00
Wangda Tan
0987a7b8cb YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue mapping. (Suma Shivaprasad via wangda) 
Change-Id: Ia1704bb8cb5070e5b180b5a85787d7b9ca57ebc6
 2017-11-16 11:25:52 -08:00
Sunil G
61ace174cd YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 2017-11-16 22:34:23 +05:30
Daniel Templeton
b246c54749 YARN-7414. FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() 
(Contributed by Soumabrata Chakraborty via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Wangda Tan
dd07038ffa YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-09 14:51:15 -08:00
Konstantinos Karanasos
ac4d2b1081 YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 2017-11-09 13:01:24 -08:00
Robert Kanter
a2c150a736 YARN-7386. Duplicate Strings in various places in Yarn memory (misha@cloudera.com via rkanter) 2017-11-09 12:12:52 -08:00
Haibo Chen
a1382a18df YARN-7388. TestAMRestart should be scheduler agnostic. 2017-11-09 10:49:50 -08:00
bibinchundatt
0a72c2f56c YARN-7454. RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup. Contributed by Jason Lowe. 2017-11-09 21:01:19 +05:30
Arun Suresh
a9c70b0e84 YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 2017-11-08 08:00:53 -08:00
Daniel Templeton
8db9d61ac2 YARN-7401. Reduce lock contention in ClusterNodeTracker#getClusterCapacity() 2017-11-07 14:53:48 -08:00
Wangda Tan
13fa2d4e3e YARN-7394. Merge code paths for Reservation/Plan queues and Auto Created queues. (Suma Shivaprasad via wangda) 2017-11-06 21:38:24 -08:00
Haibo Chen
8f214dc4f8 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 2017-11-06 15:45:37 -08:00
Jian He
a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Jian He
c723021579 YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He
673c0db43c Revert "YARN-6626. Embed REST API service into RM. Contributed by Eric Yang" 
This reverts commit 63d1084e9781e0fee876916190b69f6242dd00e4.
 2017-11-06 13:30:17 -08:00
Jian He
9e677fa05c YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He
bd96c4c235 Rebase onto latest trunk. minor conflicts 2017-11-06 13:30:13 -08:00
Billie Rinaldi
1888318c89 YARN-6903. Yarn-native-service framework core rewrite. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Eric Payne
e6ec02001f YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 2017-11-02 12:37:33 -05:00
Rohith Sharma K S
940ffe3f9c addendum patch for YARN-7289. 2017-11-02 13:55:19 +05:30
Daniel Templeton
9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00
Yufei Gu
d4811c8cfa YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 2017-10-29 16:44:16 -07:00
Daniel Templeton
e62bbbca7a YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() 2017-10-28 09:13:13 -07:00
Rohith Sharma K S
5c799ecf09 YARN-7289. Application lifetime does not work with FairScheduler. Contributed by Miklos Szegedi. 2017-10-27 22:46:38 +05:30
Sunil G
792388e1c0 YARN-7375. Possible NPE in RMWebapp when HA is enabled and the active RM fails. Contributed by Chandni Singh. 2017-10-27 20:53:57 +05:30
Wangda Tan
36e158ae98 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 
Change-Id: I14c5ea7252b7c17e86ab38f692b5f9d43196dbe0
 2017-10-26 20:15:19 -07:00
Robert Kanter
b1de78619f YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 2017-10-26 17:47:32 -07:00
Subru Krishnan
25932da6d1 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 2017-10-26 12:10:14 -07:00
Robert Kanter
2da654e34a YARN-7358. TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler (haibochen via rkanter) 2017-10-26 10:25:10 -07:00
Subru Krishnan
3fae675383 YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru). 2017-10-25 15:07:50 -07:00
Haibo Chen
d7f3737f3b YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 2017-10-24 22:17:56 -07:00
Robert Kanter
03af442e76 YARN-7385. TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE (yufeigu via rkanter) 2017-10-24 13:36:50 -07:00
Robert Kanter
025c656572 YARN-7382. NoSuchElementException in FairScheduler after failover causes RM crash (rkanter) 2017-10-24 10:21:44 -07:00
Daniel Templeton
9e77dc2bd1 YARN-7357. Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-10-23 13:51:19 -07:00
Eric Payne
921338cd86 YARN-4163: Audit getQueueInfo and getApplications calls 2017-10-23 11:43:41 -05:00
Yufei Gu
1f4cdf1068 YARN-4090. Make Collections.sort() more efficient by caching resource usage. (Contributed by Yufei Gu, Shilong Zhang and Xianyin Xin) 2017-10-20 01:32:20 -07:00
Yufei Gu
7b4b018780 YARN-7359. TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic. (Contributed by Haibo Chen) 2017-10-19 16:51:47 -07:00
Yufei Gu
cbd2b73ef8 YARN-7294. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair Scheduler. (Contributed by Miklos Szegedi) 2017-10-19 16:39:25 -07:00
Subru Krishnan
75323394fb YARN-7311. Fix TestRMWebServicesReservation parametrization for fair scheduler. (Yufei Gu via Subru). 2017-10-17 12:38:06 -07:00
Robert Kanter
8a61525928 YARN-7308. TestApplicationACLs fails with FairScheduler (rkanter) 2017-10-16 15:34:32 -07:00
Rohith Sharma K S
f4fb6695a3 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-13 15:17:05 +05:30
Robert Kanter
ec8bf9e48a YARN-7309. TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler (rkanter) 2017-10-10 10:18:42 -07:00
Wangda Tan
5cfbad68c0 YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) 
Change-Id: I959c8531bd48231956068cb7d2312f3235549f9c
(cherry picked from commit 8536db5a7549fa44d9f9e444b41bd5978b5c6ae1)
 2017-10-09 20:36:57 -07:00
Jonathan Hung
1d36b53ab6 YARN-7252. Removing queue then failing over results in exception 2017-10-09 11:11:23 -07:00
Jonathan Hung
09c5dfe937 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:11:23 -07:00
Jonathan Hung
d6622daaa3 YARN-7046. Add closing logic to configuration store 2017-10-09 11:11:23 -07:00
Wangda Tan
ff39c0de20 YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:11:23 -07:00
Xuan
4d8abd84f4 YARN-7024: Fix issues on recovery in LevelDB store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
4c8b208adb YARN-6322: Disable queue refresh when configuration mutation is enabled. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
74ba6ffa0b YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan
04f9e80bb2 YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:11:23 -07:00
Xuan
916bdbd6be YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:11:23 -07:00
Xuan
e566fd8b58 YARN-6575. Support global configuration mutation in MutableConfProvider. (Jonathan Hung via Xuan Gong) 2017-10-09 11:11:23 -07:00
Wangda Tan
a4e6253046 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:11:23 -07:00
Wangda Tan
d8a5136b1f YARN-5952. Create REST API for changing YARN scheduler configurations. (Jonathan Hung via wangda) 2017-10-09 11:11:23 -07:00
Jonathan Hung
ef686635c1 YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:11:23 -07:00
Xuan
e3579a8c3b YARN-5946: Create YarnConfigurationStore interface and 
InMemoryConfigurationStore class. Contributed by Jonathan Hung
 2017-10-09 11:11:23 -07:00
Jonathan Hung
ef59cbe08a YARN-5951. Changes to allow CapacityScheduler to use configuration store 2017-10-09 11:11:22 -07:00
Daniel Templeton
6d6ca4c923 YARN-2612 addendum: fixed javadoc error. 2017-10-09 14:12:13 +02:00
Sunil G
8137254a1f YARN-7245. Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user. Contributed by Eric Payne. 2017-10-09 14:39:41 +05:30
Rohith Sharma K S
6fe6a72352 Revert "YARN-7180. Remove class ResourceType. Contributed by Sunil G." 
This reverts commit 6bd81ab054.
 2017-10-09 13:21:13 +05:30
Rohith Sharma K S
6bd81ab054 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-09 10:20:40 +05:30
Yufei Gu
72d22b753a YARN-7207. Cache the RM proxy server address. (Yufei Gu) 2017-10-06 09:49:11 -07:00
Yufei Gu
49ae538164 YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage(addendum). (Yufei Gu) 2017-10-06 00:08:25 -07:00
Yufei Gu
99292adcef YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-05 23:51:58 -07:00
Arun Suresh
b733348dde YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 2017-10-05 09:58:04 -07:00
Subru Krishnan
e6e614e380 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-10-04 19:28:27 -07:00
Akira Ajisaka
2df1b2ac05
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails. 2017-10-05 06:07:32 +09:00
Akira Ajisaka
c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Subru Krishnan
d4d2fd1acd YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru). 2017-10-02 18:14:44 -07:00
Eric Payne
7f6118f918 YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe. 2017-09-29 10:33:57 -05:00
Jason Lowe
85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Sunil G
8facf1f976 YARN-6871. Add additional deSelects params in RMWebServices#getAppReport. Contributed by Tanuj Nayak. 2017-09-27 14:37:32 +05:30
Rohith Sharma K S
06e5a7b5cf YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R. 2017-09-26 14:34:09 +05:30
Daniel Templeton
164a063e74 YARN-6878. TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-09-22 23:53:32 +02:00
Yufei Gu
a92ef030a2 YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 2017-09-21 12:31:07 -07:00
Yufei Gu
a9019e1fb7 YARN-6499. Remove the doc about Schedulable#redistributeShare(). (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-19 18:28:31 -07:00
Arun Suresh
56ef5279c1 YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 2017-09-18 15:49:31 -07:00
Wangda Tan
e81596d06d YARN-7172. ResourceCalculator.fitsIn() should not take a cluster resource parameter. (Sen Zhao via wangda) 
Change-Id: Icc3670c9381ce7591ca69ec12da5aa52d3612d34
 2017-09-17 21:20:43 -07:00
Wangda Tan
38c14ef8d8 YARN-7149. Cross-queue preemption sometimes starves an underserved queue. (Eric Payne via wangda) 
Change-Id: Ib269991dbebce160378e8372ee6d24849c4a5ed6
(cherry picked from commit 3dfa937a1fadfc62947755872515f549b3b15e6a)
 2017-09-15 21:29:39 -07:00
Wangda Tan
958e8c0e25 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
 2017-09-15 21:29:31 -07:00
Yufei Gu
09b476e6da YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 2017-09-14 11:23:37 -07:00
Robert Kanter
bb34ae9554 YARN-7146. Many RM unit tests failing with FairScheduler (rkanter) 2017-09-13 15:44:54 -07:00