big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,351 Commits 2 Branches 2 Tags 264 MiB
cb2b7970ee
Commit Graph

2304 Commits

Author SHA1 Message Date
Szilard Nemeth
42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Jim Brennan
b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Adam Antal
bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
Szilard Nemeth
a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Shubham Gupta
e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
Eric E Payne
9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00
Szilard Nemeth
773ac799c6 YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko 2020-09-09 15:12:58 +02:00
Brahma Reddy Battula
43572fc7f8 YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 2020-09-09 17:08:13 +05:30
Peter Bacsko
c4fb4044b2 YARN-10415. Create a group matcher which checks ALL groups of the user. Contributed by Gergely Pollak. 2020-09-08 10:57:00 +02:00
Adam Antal
34fe74da0e YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 2020-09-07 11:44:09 +02:00
Peter Bacsko
360bbcd3bc YARN-10411. Create an allowCreate flag for MappingRuleAction. Contributed by Gergely Pollak. 2020-09-07 11:39:03 +02:00
Peter Bacsko
1841a5bb03 YARN-10376. Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak. 2020-09-05 10:00:33 +02:00
Szilard Nemeth
696e4fe50e Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja 2020-09-04 09:59:54 +02:00
Peter Bacsko
5c15815773 YARN-10375. CS Mapping rule config parser should return MappingRule objects. Contributed by Gergely Pollak. 2020-09-03 18:54:22 +02:00
Peter Bacsko
288dab7563 YARN-10372. Create MappingRule class to represent each CS mapping rule. Contributed by Gergely Pollak 2020-09-03 15:02:48 +02:00
Peter Bacsko
c7dab2b23e YARN-10374. Create Actions for CS mapping rules. Contributed by Gergely Pollak. 2020-09-01 17:38:49 +02:00
Adam Antal
73a0d149e2 YARN-10386. Addendum patch: Fixing ASF Warnings. Contributed by Peter Bacsko. 2020-09-01 14:39:24 +02:00
Szilard Nemeth
4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Szilard Nemeth
56a5c360a1 YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak 2020-08-29 21:34:55 +02:00
Szilard Nemeth
f4f872b778 YARN-10371. Create variable context class for CS queue mapping rules. Contributed by Gergely Pollak 2020-08-29 17:31:48 +02:00
Szilard Nemeth
6e3d7050a9 YARN-10386. Create new JSON schema for Placement Rules. Contributed by Peter Bacsko and Benjamin Teke 2020-08-28 17:09:31 +02:00
Prabhu Joseph
5c927eb550 YARN-10360. Support Multi Node Placement in SingleConstraintAppPlacementAllocator 
Reviewed by Sunil G.
 2020-08-24 15:00:02 +05:30
Zhengbo Li
9b9f7ea16a
YARN-10399 Refactor NodeQueueLoadMonitor class to make it extendable (#2228) 
Refactor NodeQueueLoadMonitor class to make it extendable
 2020-08-19 10:14:50 -07:00
Sunil G
82ec28f442 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-08-19 11:54:48 +05:30
Prabhu Joseph
6c2ce3d56b YARN-10389. Option to override RMWebServices with custom WebService class 
Contributed by Tanu Ajmera. Reviewed by Bilwa ST and Sunil G.
 2020-08-11 15:46:47 +05:30
Prabhu Joseph
5e0f879779 YARN-10364. Fix logic of queue capacity is absolute resource or percentage. 
Contributed by Bilwa ST. Reviewed by Sunil G.
 2020-08-08 22:33:58 +05:30
Jonathan Hung
64753addba YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:01 -07:00
bibinchundatt
81da221c75 YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 2020-08-07 08:36:52 +05:30
Prabhu Joseph
c7e71a6c0b YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.
 2020-08-06 13:19:52 +05:30
Eric Badger
5edd8b925e YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 18:47:49 +00:00
Prabhu Joseph
0277856738 YARN-10377. Fix filter index to show apps while clicking on queue in RM UI 
Contributed by Tarun Parimi.
 2020-08-04 23:10:45 +05:30
Prabhu Joseph
aa5afa72c5 YARN-10381. Add application attempt state in AppAttempts RM REST API 
Contributed by Siddharth Ahuja. Reviewed by Bilwa ST.
 2020-08-04 23:07:28 +05:30
bibinchundatt
5d8600e80a YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 2020-07-29 23:26:08 +05:30
Jonathan Hung
3eaf62726f YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:43:19 -07:00
bibinchundatt
5dadf963d3 YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. Contributed by Pranjal Protim Borah. 2020-07-28 11:55:47 +05:30
Prabhu Joseph
247eb0979b YARN-10319. Record Last N Scheduler Activities from ActivitiesManager 
Reviewed by Tao Yang and Adam Antal.
 2020-07-24 11:43:31 +05:30
bibinchundatt
bfcd775381 YARN-10315. Avoid sending RMNodeResourceupdate event if resource is same. Contributed by Sushil Ks. 2020-07-23 16:41:35 +05:30
Eric Badger
48f90115b5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan
 2020-07-13 23:10:39 +00:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Prabhu Joseph
3a4d05b850 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.
 2020-07-08 12:43:20 +05:30
Eric Yang
e8dc862d38 YARN-9809. Added node manager health status to resource manager registration call. 
Contributed by Eric Badger via eyang
 2020-06-30 11:39:16 -07:00
Eric E Payne
74fc13cf91 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 18:39:53 +00:00
Szilard Nemeth
6a8fd73b27 YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed by Marton Hudaky 2020-06-25 17:27:58 +02:00
Szilard Nemeth
03f855e3e7 YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko 2020-06-23 12:13:04 +02:00
Prabhu Joseph
ce1008fe61 YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios 
Contributed by Szilard Nemeth. Reviewed by Gergely Pollak.
 2020-06-22 12:49:14 +05:30
Surendra Singh Lilhore
b2facc84a1 YARN-9460. QueueACLsManager and ReservationsACLManager should not use instanceof checks. Contributed by Bilwa S T. 2020-06-20 20:09:31 +05:30
Szilard Nemeth
469841446f YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-19 14:50:24 +02:00
Szilard Nemeth
5b1a56f9f1 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:34:40 +02:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Tao Yang
7c4de59fc1 YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph. 2020-06-12 22:10:25 +08:00
Eric E Payne
56247db302 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 2020-06-09 18:43:16 +00:00
Szilard Nemeth
8abff5151a YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-05 15:00:39 +02:00
Szilard Nemeth
8b146c17b3 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-05 11:38:32 +02:00
Szilard Nemeth
b5efdea4fd YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:29:15 +02:00
Szilard Nemeth
e0a0741ac8 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-02 15:42:39 +02:00
Szilard Nemeth
593af878c0 YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-27 13:05:52 +02:00
Szilard Nemeth
fb030f3239 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit ac4540dd8e.
Reverting because of missing Jira-ID in the commit message.
 2020-05-27 13:04:48 +02:00
Jonathan Hung
c30c23cb66 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-05-26 16:29:42 -07:00
Szilard Nemeth
ac4540dd8e FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-21 13:11:44 +02:00
Prabhu Joseph
6ce295b787 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.
 2020-05-14 16:52:11 +05:30
Prabhu Joseph
450e5aa9dd YARN-10154. Addendum Patch which fixes below bugs 
1. RM fails to start when LeafQueueTemplate max capacity is not specified.
2. Job stuck in ACCEPTED state with DominantResourceCalculator as Queue
   Capacity is set to NaN during RM startup with clusterResource is zero.

Reviewed by Sunil G and Manikandan R.
 2020-05-13 14:35:37 +05:30
Jonathan Hung
fff1d2c122 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 2020-05-12 10:48:54 -07:00
Inigo Goiri
35010120fb YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 2020-05-06 13:22:54 -07:00
Szilard Nemeth
cb6399c109 YARN-10257. FS-CS converter: skip increment properties for mem/vcores and fix DRF check. Contributed by Peter Bacsko 2020-05-05 16:00:43 +02:00
Szilard Nemeth
0debe55d6c YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-05 13:11:28 +02:00
Inigo Goiri
d125d39108 YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by Bilwa S T. 2020-04-29 11:23:58 -07:00
Szilard Nemeth
410c605aec YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 2020-04-29 15:53:30 +02:00
Akira Ajisaka
6ef01646ba
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-04-28 05:35:36 +09:00
Szilard Nemeth
f91e21ac10 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 2020-04-27 08:26:11 +02:00
Szilard Nemeth
d4874585f4 YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-24 17:04:47 +02:00
Szilard Nemeth
92b28e38b7 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-18 10:13:37 +02:00
Szilard Nemeth
76900b4f5b YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-18 09:57:27 +02:00
Jonathan Hung
3601054822 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:44 -07:00
Sunil G
2fe122e322 YARN-10154. CS Dynamic Queues cannot be configured with absolute resources. Contributed by Manikandan R. 2020-04-17 18:51:19 +05:30
Szilard Nemeth
55fcbcb5c2 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 2020-04-15 07:24:04 +02:00
Jonathan Hung
23481ad378 YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 2020-04-09 11:06:11 -07:00
Sunil G
6fb29d0f17 YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 2020-04-09 18:51:42 +05:30
Szilard Nemeth
45362a9f4c YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 2020-04-07 15:51:55 +02:00
Szilard Nemeth
3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth
9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth
348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Jonathan Hung
6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
Sunil G
cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Prabhu Joseph
d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue 
Contributed by Peter Bacsko. Reviewed by Manikandan R.
 2020-03-23 18:01:14 +05:30
Prabhu Joseph
f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node 
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
 2020-03-19 12:55:14 +05:30
Weiwei Yang
b9d825f178 YARN-9567. Add diagnostics for outstanding resource requests on app attempts page. Contributed by Tao Yang. 2020-03-13 14:39:36 -07:00
Szilard Nemeth
5ead9c15ca YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-03-12 12:29:03 +01:00
Szilard Nemeth
38d87883b6 YARN-10193. FS-CS converter: fix incorrect capacity conversion. Contributed by Peter Bacsko 2020-03-12 12:18:37 +01:00
Sunil G
0fd8bf5f6b YARN-10191. FS-CS converter: call System.exit function call for every code path in main method. Contributed by Peter Bacsko. 2020-03-12 15:03:12 +05:30
Szilard Nemeth
61f4cf3055 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-03-10 16:35:04 +01:00
Szilard Nemeth
9314ef947f YARN-10168. FS-CS Converter: tool doesn't handle min/max resource conversion correctly. Contributed by Peter Bacsko 2020-03-10 16:07:46 +01:00
Szilard Nemeth
ea0444851d YARN.10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-03-06 17:49:45 +01:00
Szilard Nemeth
004e955348 YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko 2020-03-05 09:56:42 +01:00
Szilard Nemeth
c0d084247c YARN-10175. FS-CS converter: only convert placement rules if a cmd line switch is defined. Contributed by Peter Bacsko 2020-03-03 09:46:32 +01:00
Szilard Nemeth
d68616b045 YARN-10130. FS-CS converter: Do not allow output dir to be the same as input dir. Contributed by Adam Antal 2020-02-25 21:30:04 +01:00
Szilard Nemeth
72fa9c7f1b YARN-10135. FS-CS converter tool: issue warning on dynamic auto-create mapping rules. Contributed by Peter Bacsko 2020-02-24 21:54:07 +01:00
Szilard Nemeth
34cf63c7d0 YARN-10157. FS-CS converter: initPropertyActions() is not called without rules file. Contributed by Peter Bacsko 2020-02-24 21:45:49 +01:00
Sunil G
6526f95bd2 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 2020-02-19 11:17:22 +05:30
Prabhu Joseph
e637797211 YARN-10127. Remove setting App Ordering Policy to ParentQueue in FSQueueConverter 
Contributed by Peter Bacsko.
 2020-02-11 22:01:58 +05:30
Sunil G
28f730b317 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 2020-02-09 21:14:53 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Jonathan Hung
314e2f9d2e YARN-10116. Expose diagnostics in RMAppManager summary 2020-02-04 17:44:05 -08:00
Szilard Nemeth
a7d72c523a YARN-10099. FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues. Contributed by Peter Bacsko 2020-01-30 16:03:38 +01:00
Eric Badger
e578e52aae YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 03:54:43 +00:00
Inigo Goiri
5abd0148eb YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-28 10:06:37 -08:00
Prabhu Joseph
1ab9c692fa YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change 
Contributed by Kinga Marton.
 2020-01-28 23:16:04 +05:30
Szilard Nemeth
94f060242e YARN-10104. FS-CS converter: dry run should work without output defined. Contributed by Peter Bacsko 2020-01-28 15:36:04 +01:00
Szilard Nemeth
ca29768035 YARN-10085. FS-CS converter: remove mixed ordering policy check. Contributed by Peter Bacsko 2020-01-28 15:22:48 +01:00
Szilard Nemeth
9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
Inigo Goiri
b4870bce3a Revert "YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R." 
This reverts commit 0696828a09.
 2020-01-21 17:45:17 -08:00
Inigo Goiri
0696828a09 YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-21 13:41:01 -08:00
Szilard Nemeth
581072a8f0 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-20 13:10:32 +01:00
Prabhu Joseph
a0ff42d761 YARN-10070. Fix NPE if no queue mapping defined for proxy user when using application tag based placement. 
Contributed by Kinga Marton. Reviewed by Adam Antal.
 2020-01-16 12:55:27 +05:30
Szilard Nemeth
2aa065d98f YARN-10082. FS-CS converter: disable terminal placement rule checking. Contributed by Peter Bacsko 2020-01-15 12:42:13 +01:00
Szilard Nemeth
7c5cecc3b3 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-15 09:55:41 +01:00
Szilard Nemeth
1c51f36be7 YARN-9788. Queue Management API does not support parallel updates. Contributed by Prabhu Joseph 2020-01-14 12:26:03 +01:00
Szilard Nemeth
621c5eac38 YARN-9912. Capacity scheduler: support u:user2:%secondary_group queue mapping. Contributed by Manikandan R 2020-01-13 17:23:00 +01:00
Szilard Nemeth
b7ef8a333f YARN-9868. Validate %primary_group queue in CS queue manager. Contributed by Manikandan R 2020-01-13 17:15:09 +01:00
Szilard Nemeth
d842dfffa5 YARN-9866. u:user2:%primary_group is not working as expected. Contributed by Manikandan R 2020-01-12 14:04:15 +01:00
Szilard Nemeth
24e6a9e43a YARN-10067. Add dry-run feature to FS-CS converter tool. Contributed by Peter Bacsko 2020-01-12 13:48:39 +01:00
Szilard Nemeth
768ee22e9e YARN-10035. Add ability to filter the Cluster Applications API request by name. Contributed by Adam Antal 2020-01-06 16:26:33 +01:00
Prabhu Joseph
eca7e14c2f YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:13:57 +05:30
Prabhu Joseph
21ada4d1b0 Revert "YARN-10053. Use Shared Group Mapping Service in Placement Rules." 
This reverts commit 217b56ffdd.
 2020-01-02 14:12:43 +05:30
Prabhu Joseph
217b56ffdd YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:07:49 +05:30
Giovanni Matteo Fumarola
ef59ffd362 YARN-10038. [UI] Finish Time is not correctly parsed in the RM Apps page. Contributed by Inigo Goiri. 2019-12-19 11:37:17 -08:00
Jonathan Hung
fddc3d55c3 YARN-10039. Allow disabling app submission from REST endpoints 2019-12-18 10:48:05 -08:00
Jonathan Hung
6d970d8100 YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 2019-12-08 16:35:02 -08:00
Jonathan Hung
cd4779f317 Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit 92bce918dc.
 2019-12-08 16:34:46 -08:00
Jonathan Hung
92bce918dc YARN-10012. Guaranteed and max capacity queue metrics for custom resources 2019-12-08 15:52:17 -08:00
prabhujoseph
e575df5ca6 YARN-10006. IOException used in place of YARNException in CapaitySheduler 
Contributed by Adam Antal. Reviewed by Szilard Nemeth.
 2019-12-06 12:51:28 +05:30
Szilard Nemeth
cf68857631 YARN-9938. Validate Parent Queue for QueueMapping contains dynamic group as parent queue. Contributed by Manikandan R 2019-12-05 21:04:42 +01:00
Szilard Nemeth
5cc6f945da YARN-9789. Disable Option for Write Ahead Logs of LogMutation. Contributed by Prabhu Joseph 2019-12-05 20:43:26 +01:00
Szilard Nemeth
4627dd6708 YARN-9780. SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call. Contributed by Prabhu Joseph 2019-12-05 20:38:37 +01:00
Szilard Nemeth
c71befaf8f YARN-9781. SchedConfCli to get current stored scheduler configuration. Contributed by Prabhu Joseph 2019-12-05 20:31:24 +01:00
Szilard Nemeth
ef950b0863 YARN-9290. Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler. Contributed by Prabhu Joseph 2019-11-26 22:04:07 +01:00
prabhujoseph
aa7ab2719f YARN-9991. Fix Application Tag prefix to userid. Contributed by Szilard Nemeth. 2019-11-27 01:59:12 +05:30
Szilard Nemeth
8c9018d5c7 YARN-9899. Migration tool that help to generate CS config based on FS config [Phase 2]. Contributed by Peter Bacsko 2019-11-26 21:22:35 +01:00
Szilard Nemeth
448ffb12ec YARN-9937. addendum: Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2019-11-26 13:41:41 +01:00
HUAN-PING SU
f8e36e03b4 YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 2019-11-25 15:28:53 +09:00
Tao Yang
8afabe41e4 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:06:14 +08:00
Szilard Nemeth
dfdc6d6dd9 YARN-9886. Queue mapping based on userid passed through application tag. Contributed by Julia Kinga Marton 2019-11-19 21:14:15 +01:00
Prabhu Joseph
30b2096933 YARN-9986. Allow app owner to signal container via RMWebServices. Contributed by kyungwan nam. 2019-11-19 11:35:00 +00:00
Sunil G
ea68756c0c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:10:41 +05:30
Sunil G
215f2052fc YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:03:02 +05:30
Szilard Nemeth
27642367ef YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-18 16:29:39 +01:00
Yufei Gu
b83b9ab418 YARN-9537. Add configuration to disable AM preemption. Contributed by ZhouKang 2019-11-12 09:40:16 -08:00
Abhishek Modi
fb512f5087 YARN-9697. Efficient allocation of Opportunistic containers. Contributed by Abhishek Modi. 2019-11-12 16:34:04 +05:30
Szilard Nemeth
30b93f914b YARN-9865. Capacity scheduler: add support for combined %user + %secondary_group mapping. Contributed by Manikandan R 2019-11-11 13:28:02 +01:00
Sunil G
51e7d1b37e YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 2019-11-04 23:27:49 +05:30
Sunil G
d462308e04 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 2019-11-03 08:47:27 +05:30
Sunil G
9a2e43e29e YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph. 2019-10-31 00:34:27 +05:30
Szilard Nemeth
e3e7daa4f5 YARN-9889. [UI] Add Application Tag column to RM All Applications table. Contributed by Kinga Marton 2019-10-30 16:26:56 +01:00
Abhishek Modi
ed45c13f67 YARN-2442. ResourceManager JMX UI does not give HA State. Contributed by Rohith Sharma and Cyrus Jackson. 2019-10-29 15:54:55 +05:30
Eric E Payne
83d148074f YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 2019-10-21 20:30:50 +00:00
Eric Yang
3f7756dc6c YARN-9922. Fixed YARN javadoc errors from YARN-9699. 
Contributed by Peter Bacsko
 2019-10-21 16:16:49 -04:00
Eric Yang
155864da00 YARN-9875. Improve fair scheduler configuration store on HDFS. 
Contributed by Prabhu Joseph
 2019-10-18 20:30:11 -04:00
Szilard Nemeth
f0699a7406 YARN-9841. Capacity scheduler: add support for combined %user + %primary_group mapping. Contributed by Manikandan R 2019-10-18 11:19:49 +02:00
Sunil G
3990ffa083 YARN-9909. Offline format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-10-17 06:43:06 +05:30
Eric E Payne
a5034c7988 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 2019-10-16 21:10:08 +00:00
Sunil G
090f73a9aa YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 2019-10-16 18:14:31 +05:30
Szilard Nemeth
a8ef03e961 YARN-9840. Capacity scheduler: add support for Secondary Group rule mapping. Contributed by Manikandan R 2019-10-15 17:22:52 +02:00
Szilard Nemeth
5cc7873a47 YARN-9699. Migration tool that help to generate CS config based on FS config [Phase 1]. Contributed by Peter Bacsko 2019-10-14 17:19:39 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Sunil G
4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Abhishek Modi
61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
bibinchundatt
425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Sunil G
137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Jonathan Hung
c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Jonathan Hung
3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Abhishek Modi
da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
Eric Badger
56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Jonathan Hung
f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang
10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API. 
Contributed by Prabhu Joseph
 2019-09-10 17:04:39 -04:00
Jonathan Hung
387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00
Jonathan Hung
43e389b980 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:07:11 -07:00
Wangda Tan
34b82e6da0 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
 2019-09-06 19:18:13 -07:00
Tao Yang
73575701ab YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:52:39 +08:00
Jonathan Hung
b71a7f195c YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 11:00:07 -07:00
Jonathan Hung
e7d44e48f7 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:17:07 -07:00
Jonathan Hung
0ccf4b0fe1 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 2019-09-05 14:01:42 -07:00
bibinchundatt
03489124ea YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T. 2019-09-03 11:53:34 +05:30
Abhishek Modi
751b5a1ac8 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 2019-09-01 22:40:40 +05:30
Abhishek Modi
c7ef4fb74a YARN-8678. Queue Management API - rephrase error messages. Contributed by Prabhu Joseph. 2019-09-01 10:00:04 +05:30
Abhishek Modi
d2d963f3d4 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 2019-09-01 09:54:46 +05:30
Rohith Sharma K S
5b557de721 YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang. 2019-08-30 10:35:09 +05:30
Weiwei Yang
8c0759d02a YARN-9664. Improve response of scheduler/app activities for better understanding. Contributed by Tao Yang. 2019-08-29 18:14:39 +08:00
Eric E Payne
6f2226a013 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 2019-08-28 20:40:34 +00:00
Jonathan Hung
8ef46595da YARN-9438. launchTime not written to state store for running applications 2019-08-27 15:53:43 -07:00
Eric Yang
717c853873 YARN-9755. Fixed RM failing to start when FileSystemBasedConfigurationProvider is configured. 
Contributed by Prabhu Joseph
 2019-08-27 13:14:59 -04:00
Rohith Sharma K S
d70f5231a7 YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt. 2019-08-27 08:38:12 +05:30
Jonathan Hung
8660e48ca1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 2019-08-26 15:50:33 -07:00
bibinchundatt
d3ce53e507 YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt. 2019-08-26 23:21:33 +05:30
Rohith Sharma K S
689d2e6105 YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang. 2019-08-26 21:06:15 +05:30
Szilard Nemeth
7ab88dbfa6 YARN-7291. Better input parsing for resource in allocation file. Contributed by Zoltan Siegl 2019-08-21 17:01:18 +02:00