big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
22,947 Commits 2 Branches 2 Tags 264 MiB
c2731d4b63
Commit Graph

1703 Commits

Author SHA1 Message Date
Naganarasimha
eb08543c7a YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
bibinchundatt
8cf6a9a2bd YARN-7892. Revisit NodeAttribute class structure. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Naganarasimha
5dc7d6e0f3 YARN-8104. Add API to fetch node to attribute mapping. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Naganarasimha
0a01b1350d YARN-8100. Support API interface to query cluster attributes and attribute to nodes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Sunil G
b9890d1f66 YARN-7875. Node Attribute store for storing and recovering attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
bibinchundatt
a6590c1f1f YARN-8117. Fix TestRMWebServicesNodes test failure. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
bibinchundatt
901e85238d YARN-8033. CLI Integration with NodeAttributesManagerImpl. Contributed by Naganarasimha G R. 2018-09-12 16:01:00 +05:30
Sunil G
89b3ebd11e YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Sunil G
6f4bc49c6d YARN-7988. Refactor FSNodeLabelStore code for Node Attributes store support. Contributed by Bibin A Chundatt. 2018-09-12 16:01:00 +05:30
Naganarasimha
3b3b6efe21 YARN-7871. Node attributes reporting from NM to RM. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Naganarasimha
86d024ef2a YARN-7965. NodeAttributeManager add/get API is not working properly. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Eric E Payne
987d8191ad YARN-8709: CS preemption monitor always fails since one under-served queue was deleted. Contributed by Tao Yang. 2018-09-10 19:55:20 +00:00
Sunil G
1ac01444a2 YARN-8015. Support all types of placement constraint support for Capacity Scheduler. Contributed by Weiwei Yang. 2018-08-23 10:05:43 +05:30
Weiwei Yang
9c3fc3ef28 YARN-7494. Add muti-node lookup mechanism and pluggable nodes sorting policies to optimize placement decision. Contributed by Sunil Govindan. 2018-08-21 22:42:28 +08:00
Rohith Sharma K S
4aacbfff60 YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan. 2018-08-18 10:26:55 +05:30
Wei-Chiu Chuang
bdd0e01ed9 HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 18:25:38 -07:00
Weiwei Yang
0a71bf1452 YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang. 2018-08-10 14:37:45 +08:00
Weiwei Yang
08d5060605 YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang. 2018-08-10 08:32:02 +08:00
Robert Kanter
b2517dd66b YARN-4946. RM should not consider an application as COMPLETED when log aggregation is not in a terminal state (snemeth via rkanter) 2018-08-09 14:58:04 -07:00
Weiwei Yang
d352f167eb YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang. 2018-08-09 23:46:53 +08:00
Haibo Chen
889df6f194 YARN-7948. Enable fair scheduler to refresh maximum allocation for multiple resource types. (Szilard Nemeth via Haibo Chen) 2018-08-02 10:10:12 -07:00
Sunil G
12a095a496 YARN-7159. Normalize unit of resource objects in RM to avoid unit conversion in critical path. Contributed by Manikandan R. 2018-08-02 22:29:21 +05:30
Sunil G
735b492556 YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB. 2018-08-02 08:34:09 +05:30
Sunil G
a48a0cc7fd YARN-8606. Opportunistic scheduling does not work post RM failover. Contributed by Bibin A Chundatt. 2018-08-01 12:17:18 +05:30
Eric Yang
c7ebcd76bf YARN-8579. Recover NMToken of previous attempted component data. 
Contributed by Gour Saha
 2018-07-31 18:01:02 -04:00
Haibo Chen
8aa93a575e YARN-8605. TestDominantResourceFairnessPolicy.testModWhileSorting is flaky. (Wilfred Spiegelenburg via Haibo Chen) 2018-07-31 11:32:40 -07:00
Jonathan Hung
3e06a5dcea YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:44:18 -07:00
Robert Kanter
2cccf4061c YARN-8517. getContainer and getContainers ResourceManager REST API methods are not documented (snemeth via rkanter) 2018-07-27 14:35:03 -07:00
Eric Yang
f93ecf5c1e YARN-8330. Improved publishing ALLOCATED events to ATS. 
Contributed by Suma Shivaprasad
 2018-07-25 18:49:30 -04:00
Eric E Payne
9485c9aee6 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R 2018-07-25 16:22:04 +00:00
Weiwei Yang
5be9f4a5d0 YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 2018-07-25 17:35:27 +08:00
bibinchundatt
e673dd1d4d YARN-8541. RM startup failure on recovery after user deletion. Contributed by Bibin A Chundatt. 2018-07-24 23:26:59 +05:30
Sunil G
35ce6eb1f5 YARN-7748. TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events. Contributed by Weiwei Yang. 2018-07-24 22:20:17 +05:30
bibinchundatt
ff7c2eda34 YARN-8548. AllocationRespose proto setNMToken initBuilder not done. Contributed by Bilwa S T. 2018-07-24 16:17:20 +05:30
Haibo Chen
8a6bb8409c YARN-6964. Fair scheduler misuses Resources operations. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-20 10:46:28 -07:00
Weiwei Yang
cbf2026483 YARN-8528. Final states in ContainerAllocation might be modified externally causing unexpected allocation results. Contributed by Xintong Song. 2018-07-20 22:34:06 +08:00
Haibo Chen
2564884757 YARN-8436. FSParentQueue: Comparison method violates its general contract. (Wilfred Spiegelenburg via Haibo Chen) 2018-07-19 13:22:31 -07:00
Eric Yang
5836e0a46b YARN-8501. Reduce complexity of RMWebServices getApps method. 
Contributed by Szilard Nemeth
 2018-07-19 12:30:38 -04:00
Wangda Tan
a2e49f41a8 YARN-8361. Change App Name Placement Rule to use App Name instead of App Id for configuration. (Zian Chen via wangda) 
Change-Id: I17e5021f8f611a9c5e3bd4b38f25e08585afc6b1
 2018-07-16 10:57:37 -07:00
Wangda Tan
752dcce5f4 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
 2018-07-16 10:54:41 -07:00
Eric E Payne
937ef39b3f YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam 2018-07-16 16:24:21 +00:00
Robert Kanter
a129e3e74e YARN-7451. Add missing tests to verify the presence of custom resources of RM apps and scheduler webservice endpoints (snemeth via rkanter) 2018-07-06 11:04:00 -07:00
Haibo Chen
9edc74f64a YARN-7556. Fair scheduler configuration should allow resource types in the minResources and maxResources properties. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-06 11:03:48 -07:00
Anu Engineer
39ad98903a Revert "Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/hadoop into trunk" 
This reverts commit c163d1797a, reversing
changes made to 0d9804dcef.
 2018-07-05 12:22:18 -07:00
Robert Kanter
99febe7fd5 YARN-7451. Add missing tests to verify the presence of custom resources of RM apps and scheduler webservice endpoints (snemeth via rkanter) 2018-07-05 10:54:19 -07:00
Haibo Chen
1726247024 YARN-7556. Fair scheduler configuration should allow resource types in the minResources and maxResources properties. (Daniel Templeton and Szilard Nemeth via Haibo Chen) 2018-07-05 10:42:39 -07:00
Junping Du
d9ba6f3656 yarn.resourcemanager.fail-fast is used inconsistently. Contributed by Yuanbo Liu. 2018-07-03 14:46:44 +08:00
Sunil G
291194302c YARN-8379. Improve balancing resources in already satisfied queues by using Capacity Scheduler preemption. Contributed by Zian Chen. 2018-06-28 10:23:31 -07:00
Eric Yang
384764cdea YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService. 
Contributed by Chandni Singh
 2018-06-28 12:39:49 -04:00
Inigo Goiri
99948565cb YARN-8412. Move ResourceRequest.clone logic everywhere into a proper API. Contributed by Botong Huang. 2018-06-21 18:24:10 -07:00
Akira Ajisaka
652bcbb3e4
YARN-8363. Upgrade commons-lang version to 3.7 in hadoop-yarn-project. Contributed by Takanobu Asanuma. 2018-06-13 00:35:37 +09:00
Rohith Sharma K S
2df73dace0 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 2018-06-12 17:13:09 +05:30
Robert Kanter
0cd145a443 YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter) 2018-06-04 15:32:03 -07:00
Haibo Chen
ba12f87dcb YARN-8390. Fix API incompatible changes in FairScheduler's AllocationFileLoaderService. (Gergo Repas via Haibo Chen) 2018-06-04 09:15:23 -07:00
Wangda Tan
931f78718f YARN-7962. Race Condition When Stopping DelegationTokenRenewer causes RM crash during failover. (BELUGA BEHR via wangda) 
Change-Id: I617e2645f60a57080058ad5f06af860fb3f682c8
 2018-06-01 14:49:18 -07:00
Weiwei Yang
6468071f13 YARN-8367. Fix NPE in SingleConstraintAppPlacementAllocator when placement constraint in SchedulingRequest is null. Contributed by Weiwei Yang. 2018-05-31 20:46:39 +08:00
Sunil G
7c343669ba YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne. 2018-05-28 16:32:53 +05:30
Wangda Tan
f24c842d52 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda) 
Change-Id: Ieea6f3eeb83c90cd74233fea896f0fcd0f325d5f
 2018-05-25 21:53:20 -07:00
Eric E Payne
8d5509c681 YARN-8292: Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan. 2018-05-25 16:06:09 +00:00
Haibo Chen
86bc6425d4 YARN-8191. Fair scheduler: queue deletion without RM restart. (Gergo Repas via Haibo Chen) 2018-05-24 17:12:34 -07:00
Wangda Tan
bd15d2396e YARN-8290. SystemMetricsPublisher.appACLsUpdated should be invoked after application information is published to ATS to avoid "User is not set in the application report" Exception. (Eric Yang via wangda) 
Change-Id: I0ac6ddd19740d1aa7dd07111cd11af71ddc2fcaf
 2018-05-22 13:33:33 -07:00
Eric E Payne
0b4c44bdee YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 2018-05-21 20:14:58 +00:00
Haibo Chen
f48fec83d0 YARN-8248. Job hangs when a job requests a resource that its queue does not have. (Szilard Nemeth via Haibo Chen) 2018-05-21 08:10:41 -07:00
Arun Suresh
3159bffce2 YARN-7900. [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor. (Botong Huang via asuresh) 2018-05-17 20:00:52 -07:00
Haibo Chen
1f10a36021 YARN-8268. Fair scheduler: reservable queue is configured both as parent and leaf queue. (Gergo Repas via Haibo Chen) 2018-05-11 11:34:00 -07:00
Weiwei Yang
9db9cd95bd YARN-7003. DRAINING state of queues is not recovered after RM restart. Contributed by Tao Yang. 2018-05-11 10:47:04 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
d76fbbc9b8 YARN-8249. Fixed few REST APIs in RMWebServices to have static-user check. Contributed by Sunil Govindan. 2018-05-10 19:05:35 -07:00
Robert Kanter
c8b53c4364 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 2018-05-10 09:31:59 -07:00
Rohith Sharma K S
eb7b256cc2 YARN-8239. [UI2] Clicking on Node Manager UI under AM container info / App Attempt page goes to old RM UI. Contributed by Sunil G. 2018-05-09 12:00:43 +05:30
Wangda Tan
6b96a73bb0 YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda) 
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
 2018-05-08 11:34:45 -07:00
Inigo Goiri
6a69239d86 YARN-8163. Add support for Node Labels in opportunistic scheduling. Contributed by Abhishek Modi. 2018-05-04 14:59:59 -07:00
Rohith Sharma K S
ee2ce923a9 YARN-8217. RmAuthenticationFilterInitializer and TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator. Contributed by Suma Shivaprasad. 2018-05-03 14:43:40 +05:30
Inigo Goiri
e6a80e476d YARN-8151. Yarn RM Epoch should wrap around. Contributed by Young Chen. 2018-05-02 17:23:17 -07:00
Konstantinos Karanasos
9b09555451 YARN-8195. Fix constraint cardinality check in the presence of multiple target allocation tags. Contributed by Weiwei Yang. 2018-04-30 11:54:30 -07:00
Wangda Tan
92c5331423 YARN-8005. Add unit tests for queue priority with dominant resource calculator. (Zian Chen via wangda) 
Change-Id: I17a645f20869a1e5d86fa7a325c93fec908b91dc
 2018-04-27 13:08:43 -07:00
Sunil G
71220d218d YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 2018-04-27 10:42:30 +05:30
Sunil G
1634de0fc1 YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S. 2018-04-27 10:26:57 +05:30
Owen O'Malley
2adda92de1
HDFS-7240 git commit fixup 
Signed-off-by: Owen O'Malley <omalley@apache.org>
 2018-04-26 13:52:17 -07:00
Anu Engineer
4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects. 
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
 2018-04-26 05:36:04 -07:00
Robert Kanter
7ab08a9c37 HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter) 2018-04-23 15:44:15 -07:00
Konstantinos Karanasos
28e2244390 YARN-8111. Simplify PlacementConstraints API by removing allocationTagToIntraApp. Contributed by Weiwei Yang. 2018-04-20 12:24:48 -07:00
Jason Lowe
766544c0b0 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie 2018-04-20 13:22:20 -05:00
Shashikant Banerjee
e4c39f3247
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-04-19 08:35:38 +09:00
Rohith Sharma K S
241d985467 YARN-8145. yarn rmadmin -getGroups doesn't return updated groups for user. Contributed by Sunil G. 2018-04-18 13:39:41 +05:30
Yufei
d426b980ac YARN-7966. Remove method AllocationConfiguration#getQueueAcl and related unit tests. Contributed by Sen Zhao. 2018-04-17 22:53:32 -07:00
Haibo Chen
bb92bfb4ef YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 11:15:20 -07:00
Haibo Chen
86dfb70875 Revert "YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)" 
This reverts commit 0c14b6f0d8.
 2018-04-17 06:04:38 -07:00
Haibo Chen
0c14b6f0d8 YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 2018-04-17 05:59:03 -07:00
Wangda Tan
896b473f1b YARN-8138. Add unit test to validate queue priority preemption works under node partition. (Zian Chen via wangda) 
Change-Id: Ibebfab98a714c12c2dc643b6d7b9754a7f813632
(cherry picked from commit 6ee62e6b1c9b4bc3447ce870446068e626b1a492)
 2018-04-14 11:08:36 -07:00
Eric E Payne
18844599ae YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe 2018-04-12 17:53:57 +00:00
Weiwei Yang
7eb783e263 YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang. 2018-04-11 17:15:25 +08:00
Sunil G
821b0de4c5 YARN-7574. Add support for Node Labels on Auto Created Leaf Queue Template. Contributed by Suma Shivaprasad. 2018-04-09 21:17:22 +05:30
Vrushali C
345e7624d5 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 2018-04-04 15:08:03 -07:00
Konstantinos Karanasos
7853ec8d2f YARN-8013. Support application tags when defining application namespaces for placement constraints. Contributed by Weiwei Yang. 2018-04-04 10:51:58 -07:00
Haibo Chen
93d47a0ed5 YARN-8051. TestRMEmbeddedElector#testCallbackSynchronization is flaky. (Robert Kanter via Haibo Chen) 2018-04-03 07:59:20 -07:00
Wangda Tan
c78cb18c61 YARN-8091. Revisit checkUserAccessToQueue RM REST API. (wangda) 
Change-Id: I5fab3fe229c34e967487b7327c7b3c8ddf7cb795
 2018-04-02 15:22:05 -07:00
Weiwei Yang
1606dad133 YARN-7497. Add file system based scheduler configuration store. Contributed by Jiandan Yang. 2018-03-30 21:41:33 +08:00
Eric Yang
8fb00c3fce YARN-6257. Fix CapacityScheduler REST API JSON output. 
Contributed By Tao Yang
 2018-03-29 17:36:34 -04:00
Sunil G
7a59d60e0c YARN-8085. ResourceProfilesManager should be set in RMActiveServiceContext. Contributed by Tao Yang. 2018-03-29 21:41:16 +05:30
Wangda Tan
47f711eebc YARN-6629. NPE occurred when container allocation proposal is applied but its resource requests are removed before. (Tao Yang via wangda) 
Change-Id: I805880f90b3f6798ec96ed8e8e75755f390a9ad5
 2018-03-28 11:07:45 -07:00
Wangda Tan
a90471b3e6 YARN-8016. Refine PlacementRule interface and add a app-name queue mapping rule as an example. (Zian Chen via wangda) 
Change-Id: I35caf1480e0f76f5f3a53528af09312e39414bbb
 2018-03-23 16:43:40 -07:00
Wangda Tan
a08921ca6c YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (Weiwei Yang via wangda) 
Change-Id: I63b4e4192a95bf7ded98c54e46a2871c72869700
 2018-03-19 11:04:27 -07:00
Sunil G
6bd130ea45 YARN-8028. Support authorizeUserAccessToQueue in RMWebServices. Contributed by Wangda Tan. 2018-03-18 11:00:30 +05:30
Wangda Tan
b167d60763 YARN-7657. Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue. (Suma Shivaprasad via wangda) 
Change-Id: I32d566e8727840e43c0d66e39a77edef017e3a83
 2018-03-13 17:55:51 -07:00
Weiwei Yang
b451889e8e YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Tao Yang. 2018-03-08 18:13:36 +08:00
Arun Suresh
1054b48c27 YARN-7972. Support inter-app placement constraints for allocation tags by application ID. (Weiwei Yang via asuresh) 2018-03-05 11:24:17 -08:00
Arpit Agarwal
2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Yufei Gu
92cbbfe79e YARN-5028. RMStateStore should trim down app state for completed applications. Contributed by Gergo Repas. 2018-02-21 11:42:51 -08:00
Jason Lowe
9497215084 YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:06:28 -06:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Sunil G
aae629913c YARN-7292. Retrospect Resource Profile Behavior for overriding capability. Contributed by Wangda Tan. 2018-02-16 12:37:40 +05:30
Konstantinos Karanasos
0b489e564c YARN-7920. Simplify configuration for PlacementConstraints. Contributed by Wangda Tan. 2018-02-15 14:23:38 -08:00
Eric Payne
bddfe42e2c Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit c5e6e3de1c.
 2018-02-14 14:29:20 -06:00
Wangda Tan
042ef2fa7b YARN-7789. Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator. (Zian Chen via wangda) 
Change-Id: I1f6a451fe16758def6f47c046a9b8a67ad7255af
 2018-02-14 23:11:10 +08:00
Eric Payne
c5e6e3de1c YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-13 10:11:02 -06:00
Wangda Tan
d02e42cee4 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
 2018-02-12 10:29:37 +08:00
Yufei Gu
1bc03ddf97 YARN-7655. Avoid AM preemption caused by RRs for specific nodes or racks. Contributed by Steven Rand. 2018-02-08 12:32:43 -08:00
Robert Kanter
814d701d46 YARN-7841. Cleanup AllocationFileLoaderService's reloadAllocations method (snemeth via rkanter) 2018-02-06 14:36:49 -08:00
Weiwei Yang
bff858e910 YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 2018-02-06 19:33:04 +08:00
Konstantinos Karanasos
50723889cc YARN-7778. Merging of placement constraints defined at different levels. Contributed by Weiwei Yang. 2018-02-02 14:46:20 -08:00
Arun Suresh
6e5ba9366f YARN-7839. Modify PlacementAlgorithm to Check node capacity before placing request on node. (Panagiotis Garefalakis via asuresh) 2018-02-02 10:28:22 -08:00
Arun Suresh
d481344783 YARN-7822. Constraint satisfaction checker support for composite OR and AND constraints. (Weiwei Yang via asuresh) 2018-01-31 01:51:08 -08:00
Arun Suresh
add993e26a YARN-6597. Add RMContainer recovery test to verify tag population in the AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f8c5f5b237 YARN-7784. Fix Cluster metrics when placement processor is enabled. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a4c539fcdb YARN-7783. Add validation step to ensure constraints are not violated due to order in which a request is processed. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos
adbe87abf8 YARN-7788. Factor out management of temp tags from AllocationTagsManager. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Arun Suresh
8bf7c44436 YARN-7763. Allow Constraints specified in the SchedulingRequest to override application level constraints. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
28fe7f3318 YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
bdba01f73b YARN-7682. Expose canSatisfyConstraints utility function to validate a placement against a constraint. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
1efb2b6f25 YARN-6596. Introduce Placement Constraint Manager module. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
37f1a7b64f YARN-7653. Node group support for AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
88d8d3f40b YARN-7670. Modifications to the ResourceScheduler API to support SchedulingRequests. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh
801c0988b5 YARN-7522. Introduce AllocationTagsManager to associate allocation tags to nodes. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Sunil G
e9c72d04be YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 2018-01-29 20:44:38 +05:30
Wangda Tan
75618c0dec YARN-7760. [UI2] Clicking 'Master Node' or link next to 'AM Node Web UI' under application's appAttempt page goes to OLD RM UI. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ia7e2b955d3551ca43c6852fe253fd01983b56565
 2018-01-26 15:44:55 +08:00
Jason Lowe
ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Sunil G
82cc6f6968 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 2018-01-25 14:32:54 +05:30
Yufei Gu
22ee6f77e5 YARN-7755. Clean up deprecation messages for allocation increments in FS config. Contributed by Wilfred Spiegelenburg. 2018-01-22 10:30:01 -08:00
Miklos Szegedi
bc93ac229e YARN-7139. FairScheduler: finished applications are always restored to default queue. Contributed by Wilfred Spiegelenburg. 2018-01-18 17:43:47 -08:00
Sunil G
06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Yufei Gu
370f1c6283 YARN-6486. FairScheduler: Deprecate continuous scheduling. (Contributed by Wilfred Spiegelenburg) 2018-01-16 14:46:55 -08:00
Rohith Sharma K S
d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Sunil G
afd8caba27 YARN-7722. Rename variables in MockNM, MockRM for better clarity. Contributed by Lovekesh bansal 2018-01-10 20:51:34 +05:30
Jason Lowe
90c86d6e09 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 2018-01-09 10:51:15 -06:00
Jason Lowe
ebff4decfe YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 2018-01-09 09:52:59 -06:00
Ray Chiang
59ab5da0a0 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2018-01-08 16:19:02 -08:00
Wangda Tan
a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter
2aa4f0a559 YARN-7645. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers is flakey with FairScheduler (rkanter) 2018-01-05 13:55:09 -08:00
Robert Kanter
f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Sunil G
0c75d0634b YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:42:17 +05:30
Robert Kanter
7a55044803 YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-03 15:31:50 -08:00
Robert Kanter
382215c72b YARN-7577. Unit Fail: TestAMRestart#testPreemptedAMRestartOnRMRestart (miklos.szegedi@cloudera.com via rkanter) 2017-12-20 13:39:00 -08:00
Eric Yang
94a2ac6b71 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 18:42:27 -05:00
Sunil G
890d3d0645 YARN-7638. Unit tests related to preemption for auto created leaf queues feature.Contributed by Suma Shivaprasad. 2017-12-15 13:00:57 +05:30
Sunil G
cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Sunil G
8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G
5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G
312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Wangda Tan
a8316df8c0 YARN-7520. Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues. (Suma Shivaprasad via wangda) 
Change-Id: I482f086945bd448d512cb5b3879d7371e37ee134
 2017-12-08 15:11:28 -08:00
Wangda Tan
b38643c9a8 YARN-7473. Implement Framework and policy for capacity management of auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: Icca7805fe12f6f7fb335effff4b121b6f7f6337b
 2017-12-08 15:10:16 -08:00
Wangda Tan
74665e3a7d YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
 2017-12-08 15:07:56 -08:00
Sunil G
4db4a4a165 YARN-7575. NPE in scheduler UI when max-capacity is not configured. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G
daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan
1012b901c8 YARN-7544. Use queue-path.capacity/maximum-capacity to specify absolute min/max resources. (Sunil G via wangda) 
Change-Id: I685341be213eee500f51e02f01c91def89391c17
 2017-12-07 18:56:54 -08:00
Wangda Tan
b7b8cd5324 YARN-7538. Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring. (Sunil G via wangda) 
Change-Id: Ic9bd7e599c56970fe01cb0e1bba6df7d1f77eb29
 2017-12-07 18:56:54 -08:00
Wangda Tan
7462c38277 YARN-7483. CapacityScheduler test cases cleanup post YARN-5881. (Sunil G via wangda) 
Change-Id: I9741a6baf5cb7352d05636efb6c0b24790e7589a
 2017-12-07 18:56:54 -08:00
Wangda Tan
034b312d9f YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
 2017-12-07 18:56:54 -08:00
Wangda Tan
aa3f62740f YARN-7332. Compute effectiveCapacity per each resource vector. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan
5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda) 
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
 2017-12-07 18:56:54 -08:00
Sunil G
a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G
f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Robert Kanter
d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Arun Suresh
37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Robert Kanter
c83fe44917 YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 2017-12-01 12:18:13 -08:00
Wangda Tan
a63d19d365 YARN-6124. Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues. (Zian Chen via wangda) 
Change-Id: Id93656f3af7dcd78cafa94e33663c78d410d43c2
 2017-11-30 15:57:22 -08:00
Daniel Templeton
8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Jian He
fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Yufei Gu
2bde3aedf1 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 2017-11-24 23:32:46 -08:00
Wangda Tan
834e91ee91 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
 2017-11-23 19:59:03 -08:00
Arun Suresh
b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Yufei Gu
4cc9479dae YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 2017-11-22 14:18:36 -08:00
Wangda Tan
0d781dd03b YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
 2017-11-20 11:48:15 -08:00
Wangda Tan
0987a7b8cb YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue mapping. (Suma Shivaprasad via wangda) 
Change-Id: Ia1704bb8cb5070e5b180b5a85787d7b9ca57ebc6
 2017-11-16 11:25:52 -08:00
Sunil G
61ace174cd YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 2017-11-16 22:34:23 +05:30
Konstantinos Karanasos
ac4d2b1081 YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 2017-11-09 13:01:24 -08:00
Haibo Chen
a1382a18df YARN-7388. TestAMRestart should be scheduler agnostic. 2017-11-09 10:49:50 -08:00
Wangda Tan
13fa2d4e3e YARN-7394. Merge code paths for Reservation/Plan queues and Auto Created queues. (Suma Shivaprasad via wangda) 2017-11-06 21:38:24 -08:00
Haibo Chen
8f214dc4f8 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 2017-11-06 15:45:37 -08:00
Jian He
a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Eric Payne
e6ec02001f YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 2017-11-02 12:37:33 -05:00
Rohith Sharma K S
940ffe3f9c addendum patch for YARN-7289. 2017-11-02 13:55:19 +05:30
Daniel Templeton
9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00