big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,101 Commits 2 Branches 2 Tags 264 MiB
4031b0774e
Commit Graph

4537 Commits

Author SHA1 Message Date
Szilard Nemeth
bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt
61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Neil
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Eric Badger
4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.
 2020-12-21 23:10:31 +00:00
zhuqi
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. (#2563) 
Qi Zhu via Zhankun Tang
 2020-12-21 10:04:16 +08:00
Akira Ajisaka
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka 2020-12-16 17:26:15 +01:00
Szilard Nemeth
df35c7f519 YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak 2020-12-15 17:30:14 +01:00
adamantal
3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
adamantal
0a45bd034e YARN-10520. Deprecated the residual nested class for the LCEResourceHandler (#2527). Contributed by Wanqiang Ji 2020-12-09 12:55:24 +01:00
zhuqi
d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler (#2494) 
Contributed by Qi Zhu.
 2020-12-09 19:48:39 +08:00
Eric Badger
7d3c8ef606 YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.
 2020-12-08 00:53:24 +00:00
adamantal
9969745343 YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy 2020-12-03 09:55:06 +01:00
Eric Payne
fa773a8326 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-01 22:06:47 +00:00
Ankit Kumar
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-25 13:00:09 +09:00
Ahmed Hussein
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions (#2461) 2020-11-24 11:09:30 -08:00
Szilard Nemeth
0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth
5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Peter Bacsko
567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
Eric E Payne
55339c2bdd YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 2020-11-05 21:50:46 +00:00
Eric E Payne
0461a07c01 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 16:55:06 +00:00
Surendra Singh Lilhore
7169ec4509
YARN-10442. RM should make sure node label file highly available. (#2390) 
* YARN-10442. RM should make sure node label file highly available.

* YARN-10442. Corrected method name.
            Added license header for file TestNodeLabelFileReplication.java
            Added Test code.

* YARN-10442. Changed property name and log.

* YARN-10442. Changed default replication for FS store file.
 2020-10-30 23:05:16 +05:30
Wangda Tan
c47c9fd65d YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. (Peter Bacsko via wangda) 
Change-Id: I518dc925187ce55e9d35a37ba20878c0f4e37e5c
 2020-10-30 09:49:48 -07:00
Jim Brennan
8ee6bc2518 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 19:49:18 +00:00
Jonathan Hung
bab5bf9743 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:33:18 -07:00
Sunil G
7435604a91 YARN-10453. Add partition resource info to get-node-labels and label-mappings api responses. Contributed Akhil PB. 2020-10-22 12:52:58 +05:30
Szilard Nemeth
31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Eric Badger
8b8c672780 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-15 17:57:43 +00:00
Szilard Nemeth
42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Jim Brennan
deb35a32ba YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:46:47 +00:00
Jim Brennan
b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Adam Antal
a1f7e760df YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-05 10:09:14 +02:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Adam Antal
bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
Szilard Nemeth
a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Shubham Gupta
e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
Szilard Nemeth
eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Jim Brennan
90894ea641 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-14 21:19:14 +00:00
Eric E Payne
9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00
Szilard Nemeth
773ac799c6 YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko 2020-09-09 15:12:58 +02:00
Brahma Reddy Battula
43572fc7f8 YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 2020-09-09 17:08:13 +05:30
Peter Bacsko
aba4a506d6 YARN-10424. Adapt existing AppName and UserGroupMapping unittests to ensure backwards compatibility. Contributed by Benjamin Teke. 2020-09-09 09:41:11 +02:00
Peter Bacsko
c4fb4044b2 YARN-10415. Create a group matcher which checks ALL groups of the user. Contributed by Gergely Pollak. 2020-09-08 10:57:00 +02:00
Adam Antal
34fe74da0e YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 2020-09-07 11:44:09 +02:00
Peter Bacsko
360bbcd3bc YARN-10411. Create an allowCreate flag for MappingRuleAction. Contributed by Gergely Pollak. 2020-09-07 11:39:03 +02:00
Peter Bacsko
1841a5bb03 YARN-10376. Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak. 2020-09-05 10:00:33 +02:00
Szilard Nemeth
696e4fe50e Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja 2020-09-04 09:59:54 +02:00
Peter Bacsko
5c15815773 YARN-10375. CS Mapping rule config parser should return MappingRule objects. Contributed by Gergely Pollak. 2020-09-03 18:54:22 +02:00
Peter Bacsko
288dab7563 YARN-10372. Create MappingRule class to represent each CS mapping rule. Contributed by Gergely Pollak 2020-09-03 15:02:48 +02:00
Masatake Iwasaki
5e12dc51f0
YARN-10419. Javadoc error in hadoop-yarn-server-common module. (#2271) 2020-09-03 16:17:52 +09:00
Peter Bacsko
c7dab2b23e YARN-10374. Create Actions for CS mapping rules. Contributed by Gergely Pollak. 2020-09-01 17:38:49 +02:00
Adam Antal
73a0d149e2 YARN-10386. Addendum patch: Fixing ASF Warnings. Contributed by Peter Bacsko. 2020-09-01 14:39:24 +02:00
Szilard Nemeth
4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Szilard Nemeth
56a5c360a1 YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak 2020-08-29 21:34:55 +02:00
Szilard Nemeth
f4f872b778 YARN-10371. Create variable context class for CS queue mapping rules. Contributed by Gergely Pollak 2020-08-29 17:31:48 +02:00
Szilard Nemeth
a888d580d8 YARN-10409. Improve MockQueueHierarchyBuilder to detect queue ambiguity. Contributed by Gergely Pollak 2020-08-29 15:24:50 +02:00
Szilard Nemeth
44542863f4 YARN-10408. Extract MockQueueHierarchyBuilder to a separate class. Contributed by Gergely Pollak 2020-08-28 17:35:45 +02:00
Szilard Nemeth
6e3d7050a9 YARN-10386. Create new JSON schema for Placement Rules. Contributed by Peter Bacsko and Benjamin Teke 2020-08-28 17:09:31 +02:00
Adam Antal
82a7505646 YARN-10304. Create an endpoint for remote application log directory path query. Contributed by Andras Gyori 2020-08-25 13:29:12 +02:00
Prabhu Joseph
5c927eb550 YARN-10360. Support Multi Node Placement in SingleConstraintAppPlacementAllocator 
Reviewed by Sunil G.
 2020-08-24 15:00:02 +05:30
Zhengbo Li
9b9f7ea16a
YARN-10399 Refactor NodeQueueLoadMonitor class to make it extendable (#2228) 
Refactor NodeQueueLoadMonitor class to make it extendable
 2020-08-19 10:14:50 -07:00
Sunil G
82ec28f442 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-08-19 11:54:48 +05:30
Jim Brennan
4a7deae478 YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger 2020-08-17 16:41:15 +00:00
hemanthboyina
86bbd38c8d YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T. 2020-08-13 23:44:42 +05:30
Prabhu Joseph
6c2ce3d56b YARN-10389. Option to override RMWebServices with custom WebService class 
Contributed by Tanu Ajmera. Reviewed by Bilwa ST and Sunil G.
 2020-08-11 15:46:47 +05:30
Prabhu Joseph
5e0f879779 YARN-10364. Fix logic of queue capacity is absolute resource or percentage. 
Contributed by Bilwa ST. Reviewed by Sunil G.
 2020-08-08 22:33:58 +05:30
Jonathan Hung
64753addba YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:01 -07:00
bibinchundatt
81da221c75 YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 2020-08-07 08:36:52 +05:30
Prabhu Joseph
c7e71a6c0b YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.
 2020-08-06 13:19:52 +05:30
Eric Badger
5edd8b925e YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 18:47:49 +00:00
Prabhu Joseph
0277856738 YARN-10377. Fix filter index to show apps while clicking on queue in RM UI 
Contributed by Tarun Parimi.
 2020-08-04 23:10:45 +05:30
Prabhu Joseph
aa5afa72c5 YARN-10381. Add application attempt state in AppAttempts RM REST API 
Contributed by Siddharth Ahuja. Reviewed by Bilwa ST.
 2020-08-04 23:07:28 +05:30
Brahma Reddy Battula
eac558380f YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 2020-08-03 12:54:36 +05:30
bibinchundatt
5323e83edf YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 2020-08-01 13:03:46 +05:30
Eric E Payne
e0c9653166 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 15:30:22 +00:00
bibinchundatt
5d8600e80a YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 2020-07-29 23:26:08 +05:30
Jonathan Hung
3eaf62726f YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:43:19 -07:00
bibinchundatt
5dadf963d3 YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. Contributed by Pranjal Protim Borah. 2020-07-28 11:55:47 +05:30
Akira Ajisaka
d02be17a26
YARN-10362. Javadoc for TimelineReaderAuthenticationFilterInitializer is broken. Contributed by Xieming Li. 2020-07-27 01:55:04 +09:00
Eric Badger
ac5f21dbef YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.
 2020-07-24 22:35:16 +00:00
Prabhu Joseph
247eb0979b YARN-10319. Record Last N Scheduler Activities from ActivitiesManager 
Reviewed by Tao Yang and Adam Antal.
 2020-07-24 11:43:31 +05:30
bibinchundatt
bfcd775381 YARN-10315. Avoid sending RMNodeResourceupdate event if resource is same. Contributed by Sushil Ks. 2020-07-23 16:41:35 +05:30
Masatake Iwasaki
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Eric Badger
736bed6d6d [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan
 2020-07-20 18:08:27 +00:00
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Prabhu Joseph
cc71d50b21 YARN-10339. Fix TimelineClient in NodeManager failing when Simple Http Auth used in Secure Cluster 
Contributed by Tarun Parimi.
 2020-07-16 23:36:49 +05:30
Ahmed Hussein
1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 11:39:06 -05:00
Bilwa S T
8980c23fc2
YARN-10350. TestUserGroupMappingPlacementRule fails 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-07-15 14:31:34 +09:00
Eric Badger
48f90115b5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan
 2020-07-13 23:10:39 +00:00
Eric E Payne
0427100b75 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-07-13 18:57:50 +00:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Prabhu Joseph
3a4d05b850 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.
 2020-07-08 12:43:20 +05:30
Prabhu Joseph
2bbd00dff4 YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.
 2020-07-07 17:10:59 +05:30
Akira Ajisaka
639acb6d89
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein. 2020-07-06 16:08:36 +09:00
Szilard Nemeth
04abd0eb17 YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 14:10:55 +02:00
Eric Yang
e8dc862d38 YARN-9809. Added node manager health status to resource manager registration call. 
Contributed by Eric Badger via eyang
 2020-06-30 11:39:16 -07:00
Eric E Payne
74fc13cf91 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 18:39:53 +00:00
Brahma Reddy Battula
2c03524fa4 YARN-6526. Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call. COntributed by Bilwa S T. 2020-06-26 20:43:27 +05:30
Szilard Nemeth
6459cc73e6 YARN-10318. ApplicationHistory Web UI incorrect column indexing. Contributed by Andras Gyori 2020-06-26 13:15:14 +02:00
Szilard Nemeth
33d3df7ee2 YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy. Contributed by Szilard Nemeth 2020-06-26 11:14:24 +02:00
Szilard Nemeth
6a8fd73b27 YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed by Marton Hudaky 2020-06-25 17:27:58 +02:00
Szilard Nemeth
03f855e3e7 YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko 2020-06-23 12:13:04 +02:00
Prabhu Joseph
ce1008fe61 YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios 
Contributed by Szilard Nemeth. Reviewed by Gergely Pollak.
 2020-06-22 12:49:14 +05:30
Surendra Singh Lilhore
b2facc84a1 YARN-9460. QueueACLsManager and ReservationsACLManager should not use instanceof checks. Contributed by Bilwa S T. 2020-06-20 20:09:31 +05:30
Szilard Nemeth
469841446f YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-19 14:50:24 +02:00
Szilard Nemeth
5b1a56f9f1 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:34:40 +02:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Tao Yang
7c4de59fc1 YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph. 2020-06-12 22:10:25 +08:00
Eric E Payne
56247db302 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 2020-06-09 18:43:16 +00:00
Szilard Nemeth
8abff5151a YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-05 15:00:39 +02:00
Szilard Nemeth
8b146c17b3 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-05 11:38:32 +02:00
Szilard Nemeth
b5efdea4fd YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:29:15 +02:00
Szilard Nemeth
aa6d13455b YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-02 15:53:48 +02:00
Szilard Nemeth
e0a0741ac8 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-02 15:42:39 +02:00
Szilard Nemeth
593af878c0 YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-27 13:05:52 +02:00
Szilard Nemeth
fb030f3239 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit ac4540dd8e.
Reverting because of missing Jira-ID in the commit message.
 2020-05-27 13:04:48 +02:00
Jonathan Hung
c30c23cb66 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-05-26 16:29:42 -07:00
Szilard Nemeth
ac4540dd8e FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-21 13:11:44 +02:00
Prabhu Joseph
cef0756929 YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient. 
Contributed by Bilwa S T. Reviewed by Akira Ajisaka.
 2020-05-20 11:36:52 +05:30
Prabhu Joseph
6ce295b787 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.
 2020-05-14 16:52:11 +05:30
Inigo Goiri
108ecf992f YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 2020-05-13 10:04:12 -07:00
Prabhu Joseph
450e5aa9dd YARN-10154. Addendum Patch which fixes below bugs 
1. RM fails to start when LeafQueueTemplate max capacity is not specified.
2. Job stuck in ACCEPTED state with DominantResourceCalculator as Queue
   Capacity is set to NaN during RM startup with clusterResource is zero.

Reviewed by Sunil G and Manikandan R.
 2020-05-13 14:35:37 +05:30
Jonathan Hung
fff1d2c122 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 2020-05-12 10:48:54 -07:00
Inigo Goiri
35010120fb YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 2020-05-06 13:22:54 -07:00
Ahmed Hussein
92e3ebb401 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-05-06 12:27:17 -05:00
Szilard Nemeth
cb6399c109 YARN-10257. FS-CS converter: skip increment properties for mem/vcores and fix DRF check. Contributed by Peter Bacsko 2020-05-05 16:00:43 +02:00
Szilard Nemeth
0debe55d6c YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-05 13:11:28 +02:00
Ahmed Hussein
f5081a9a5d YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-05-04 10:13:51 -05:00
Surendra Singh Lilhore
257b04ce4e YARN-6553. Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests. Contributed by Bilwa S T. 2020-05-02 12:43:20 +05:30
Inigo Goiri
d125d39108 YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by Bilwa S T. 2020-04-29 11:23:58 -07:00
Szilard Nemeth
410c605aec YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 2020-04-29 15:53:30 +02:00
Szilard Nemeth
ab36429559 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 2020-04-28 18:53:28 +02:00
Akira Ajisaka
6ef01646ba
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-04-28 05:35:36 +09:00
Szilard Nemeth
f91e21ac10 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 2020-04-27 08:26:11 +02:00
Szilard Nemeth
d4874585f4 YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-24 17:04:47 +02:00
Akira Ajisaka
9827ff2961
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 2020-04-24 09:32:25 +09:00
Szilard Nemeth
92b28e38b7 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-18 10:13:37 +02:00
Szilard Nemeth
76900b4f5b YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-18 09:57:27 +02:00
Jonathan Hung
3601054822 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:44 -07:00
Sunil G
2fe122e322 YARN-10154. CS Dynamic Queues cannot be configured with absolute resources. Contributed by Manikandan R. 2020-04-17 18:51:19 +05:30
Szilard Nemeth
4db598e0e6 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-15 13:18:44 +02:00
Szilard Nemeth
55fcbcb5c2 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 2020-04-15 07:24:04 +02:00
Jonathan Hung
23481ad378 YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 2020-04-09 11:06:11 -07:00
Sunil G
6fb29d0f17 YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 2020-04-09 18:51:42 +05:30
Wilfred Spiegelenburg
2214005c0f
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 2020-04-08 11:30:03 +10:00
Szilard Nemeth
45362a9f4c YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 2020-04-07 15:51:55 +02:00
Prabhu Joseph
7a39da3537 YARN-10120. Amendment fix for Java Doc. 
Contributed by Bilwa S T.
 2020-04-03 13:07:41 +05:30
Prabhu Joseph
25361b077b YARN-10120. Https Support in Router WebServiceClient. 
Contributed by Bilwa S T.
 2020-04-02 18:11:43 +05:30
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Szilard Nemeth
3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth
9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth
348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Jonathan Hung
6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
Sunil G
cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Prabhu Joseph
d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue 
Contributed by Peter Bacsko. Reviewed by Manikandan R.
 2020-03-23 18:01:14 +05:30
Prabhu Joseph
f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node 
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
 2020-03-19 12:55:14 +05:30
Weiwei Yang
b9d825f178 YARN-9567. Add diagnostics for outstanding resource requests on app attempts page. Contributed by Tao Yang. 2020-03-13 14:39:36 -07:00
Bilwa
c83644deac YARN-10110. Adding RouterPolicyProvider for RM Federation. 
Contributed by Bilwa S T.
 2020-03-13 16:56:44 +05:30
Szilard Nemeth
5ead9c15ca YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-03-12 12:29:03 +01:00
Szilard Nemeth
38d87883b6 YARN-10193. FS-CS converter: fix incorrect capacity conversion. Contributed by Peter Bacsko 2020-03-12 12:18:37 +01:00
Sunil G
0fd8bf5f6b YARN-10191. FS-CS converter: call System.exit function call for every code path in main method. Contributed by Peter Bacsko. 2020-03-12 15:03:12 +05:30
Szilard Nemeth
cf9cf83a43 YARN-9354. Resources should be created with ResourceTypesTestHelper instead of TestUtils. Contributed by Andras Gyori 2020-03-10 16:44:48 +01:00
Szilard Nemeth
61f4cf3055 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-03-10 16:35:04 +01:00
Szilard Nemeth
9314ef947f YARN-10168. FS-CS Converter: tool doesn't handle min/max resource conversion correctly. Contributed by Peter Bacsko 2020-03-10 16:07:46 +01:00
Eric E Payne
ede05b19d1 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 2020-03-10 14:17:45 +00:00
Szilard Nemeth
44afe1154d YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-09 16:08:24 +01:00
bibinchundatt
3859fa76d0 YARN-6924. Metrics for Federation AMRMProxy. Contributed by Young Chen 2020-03-07 09:34:42 +05:30
Szilard Nemeth
ea0444851d YARN.10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-03-06 17:49:45 +01:00
Szilard Nemeth
004e955348 YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko 2020-03-05 09:56:42 +01:00
Eric Badger
2649f8b327 YARN-10173. Make pid file generation timeout configurable in case of reacquired 
container. Contributed by Adam Antal.
 2020-03-04 23:31:57 +00:00
Szilard Nemeth
c0d084247c YARN-10175. FS-CS converter: only convert placement rules if a cmd line switch is defined. Contributed by Peter Bacsko 2020-03-03 09:46:32 +01:00
Szilard Nemeth
e9eecedf69 YARN-10148. addendum: Fix method call parameter order of setAdminAndSubmitACL in TestCapacitySchedulerQueueACLs. Contributed by Kinga Marton 2020-03-02 16:25:14 +01:00
Inigo Goiri
a43510e21d YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 2020-02-27 13:18:30 -08:00
Inigo Goiri
b420ddeada YARN-10155. TestDelegationTokenRenewer.testTokenThreadTimeout fails in trunk. Contributed by Manikandan R. 2020-02-27 12:53:20 -08:00
Szilard Nemeth
10461e0193 YARN-10148. Add Unit test for queue ACL for both FS and CS. Contributed by Kinga Marton 2020-02-27 20:10:32 +01:00
Brahma Reddy Battula
3a9ccf7f6d YARN-10141.Interceptor in FederationInterceptorREST doesnt update on RM switchover. Contributed by D M Murali Krishna Reddy. 2020-02-26 23:24:00 +05:30
Szilard Nemeth
d68616b045 YARN-10130. FS-CS converter: Do not allow output dir to be the same as input dir. Contributed by Adam Antal 2020-02-25 21:30:04 +01:00
Szilard Nemeth
72fa9c7f1b YARN-10135. FS-CS converter tool: issue warning on dynamic auto-create mapping rules. Contributed by Peter Bacsko 2020-02-24 21:54:07 +01:00
Szilard Nemeth
34cf63c7d0 YARN-10157. FS-CS converter: initPropertyActions() is not called without rules file. Contributed by Peter Bacsko 2020-02-24 21:45:49 +01:00
Szilard Nemeth
3f1aad05f0 YARN-10147. FPGA plugin can't find the localized aocx file. Contributed by Peter Bacsko 2020-02-19 15:54:25 +01:00
Sunil G
6526f95bd2 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 2020-02-19 11:17:22 +05:30
Akira Ajisaka
d6d7f8d8c5
YARN-8374. Upgrade objenesis to 2.6 (#1798) 2020-02-19 09:50:37 +09:00
Brahma Reddy Battula
20add89718 YARN-10136. [Router] : Application metrics are hardcode as N/A in UI. Contributed by Bilwa S T. 2020-02-14 16:50:29 +05:30
Prabhu Joseph
e637797211 YARN-10127. Remove setting App Ordering Policy to ParentQueue in FSQueueConverter 
Contributed by Peter Bacsko.
 2020-02-11 22:01:58 +05:30
Sunil G
28f730b317 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 2020-02-09 21:14:53 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth
71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Jonathan Hung
314e2f9d2e YARN-10116. Expose diagnostics in RMAppManager summary 2020-02-04 17:44:05 -08:00
Giovanni Matteo Fumarola
bf8686f43f YARN-8982. [Router] Add locality policy. Contributed by Young Chen. 2020-01-30 16:59:36 -08:00
Szilard Nemeth
a7d72c523a YARN-10099. FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues. Contributed by Peter Bacsko 2020-01-30 16:03:38 +01:00
Akira Ajisaka
a5ef08b619
YARN-9743. [JDK11] TestTimelineWebServices.testContextFactory fails. (#1824) Contributed by Akira Ajisaka and Kinga Marton. 2020-01-30 14:10:31 +09:00
Prabhu Joseph
825db8fe2a YARN-10107. Fix GpuResourcePlugin#getNMResourceInfo to honor Auto Discovery Enabled 
Contributed by Szilard Nemeth.
 2020-01-29 13:30:00 +05:30
Eric Badger
e578e52aae YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 03:54:43 +00:00
Inigo Goiri
5abd0148eb YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-28 10:06:37 -08:00
Prabhu Joseph
1ab9c692fa YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change 
Contributed by Kinga Marton.
 2020-01-28 23:16:04 +05:30
Szilard Nemeth
94f060242e YARN-10104. FS-CS converter: dry run should work without output defined. Contributed by Peter Bacsko 2020-01-28 15:36:04 +01:00
Szilard Nemeth
ca29768035 YARN-10085. FS-CS converter: remove mixed ordering policy check. Contributed by Peter Bacsko 2020-01-28 15:22:48 +01:00
Szilard Nemeth
9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
Inigo Goiri
b4870bce3a Revert "YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R." 
This reverts commit 0696828a09.
 2020-01-21 17:45:17 -08:00
Inigo Goiri
0696828a09 YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-21 13:41:01 -08:00
Szilard Nemeth
581072a8f0 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-20 13:10:32 +01:00
Szilard Nemeth
8b3ee2f7e9 YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-20 09:54:22 +01:00
Prabhu Joseph
a0ff42d761 YARN-10070. Fix NPE if no queue mapping defined for proxy user when using application tag based placement. 
Contributed by Kinga Marton. Reviewed by Adam Antal.
 2020-01-16 12:55:27 +05:30
Akira Ajisaka
14c2c3d69d
YARN-9512. [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException. (#1806) 2020-01-16 12:21:50 +09:00
Szilard Nemeth
2aa065d98f YARN-10082. FS-CS converter: disable terminal placement rule checking. Contributed by Peter Bacsko 2020-01-15 12:42:13 +01:00
Szilard Nemeth
7c5cecc3b3 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-15 09:55:41 +01:00
Szilard Nemeth
1c51f36be7 YARN-9788. Queue Management API does not support parallel updates. Contributed by Prabhu Joseph 2020-01-14 12:26:03 +01:00
Szilard Nemeth
13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00
Szilard Nemeth
621c5eac38 YARN-9912. Capacity scheduler: support u:user2:%secondary_group queue mapping. Contributed by Manikandan R 2020-01-13 17:23:00 +01:00
Szilard Nemeth
b7ef8a333f YARN-9868. Validate %primary_group queue in CS queue manager. Contributed by Manikandan R 2020-01-13 17:15:09 +01:00
Szilard Nemeth
d842dfffa5 YARN-9866. u:user2:%primary_group is not working as expected. Contributed by Manikandan R 2020-01-12 14:04:15 +01:00
Szilard Nemeth
24e6a9e43a YARN-10067. Add dry-run feature to FS-CS converter tool. Contributed by Peter Bacsko 2020-01-12 13:48:39 +01:00
Eric Yang
2301b25899 YARN-10019. Improved container-executor exec() calls. 
Contributed by Peter Bacsko
 2020-01-10 19:04:04 -05:00
Eric E Payne
93233a7d6e YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 2020-01-09 17:18:44 +00:00
Eric E Payne
b1e07d27cc YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 19:26:01 +00:00
Eric E Payne
6899be5a17 YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 17:29:56 +00:00
Szilard Nemeth
dd2607e3ec YARN-10026. Pull out common code pieces from ATS v1.5 and v2. Contributed by Adam Antal 2020-01-06 17:16:11 +01:00
Szilard Nemeth
768ee22e9e YARN-10035. Add ability to filter the Cluster Applications API request by name. Contributed by Adam Antal 2020-01-06 16:26:33 +01:00
Prabhu Joseph
eca7e14c2f YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:13:57 +05:30
Prabhu Joseph
21ada4d1b0 Revert "YARN-10053. Use Shared Group Mapping Service in Placement Rules." 
This reverts commit 217b56ffdd.
 2020-01-02 14:12:43 +05:30
Prabhu Joseph
217b56ffdd YARN-10053. Use Shared Group Mapping Service in Placement Rules. 
Contributed by Wilfred Spiegelenburg.
 2020-01-02 14:07:49 +05:30
Eric Badger
412035b47a YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 18:07:00 +00:00
Giovanni Matteo Fumarola
ef59ffd362 YARN-10038. [UI] Finish Time is not correctly parsed in the RM Apps page. Contributed by Inigo Goiri. 2019-12-19 11:37:17 -08:00
Jonathan Hung
7b93575b92 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 2019-12-18 13:18:11 -08:00
Jonathan Hung
fddc3d55c3 YARN-10039. Allow disabling app submission from REST endpoints 2019-12-18 10:48:05 -08:00
Eric Badger
f47dcf2d4c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 2019-12-17 17:11:23 +00:00
Szilard Nemeth
631dbbc6f2 YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal 2019-12-15 17:28:04 +01:00
prabhujoseph
d4bde134e3 YARN-7721. Fix TestContinuousScheduling fails sporadically with NPE. 
Contributed by Wilfred Spiegelenburg.
 2019-12-10 15:00:23 +05:30
Eric Badger
8dffd8dc89 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-09 01:25:10 +00:00
Eric Badger
ab2cc5ac92 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger" 
This reverts commit 76bb297db9.
 2019-12-09 01:25:10 +00:00
Jonathan Hung
6d970d8100 YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 2019-12-08 16:35:02 -08:00
Jonathan Hung
cd4779f317 Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit 92bce918dc.
 2019-12-08 16:34:46 -08:00
Jonathan Hung
92bce918dc YARN-10012. Guaranteed and max capacity queue metrics for custom resources 2019-12-08 15:52:17 -08:00
Eric Badger
76bb297db9 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-06 18:33:58 +00:00
prabhujoseph
e575df5ca6 YARN-10006. IOException used in place of YARNException in CapaitySheduler 
Contributed by Adam Antal. Reviewed by Szilard Nemeth.
 2019-12-06 12:51:28 +05:30
Szilard Nemeth
cf68857631 YARN-9938. Validate Parent Queue for QueueMapping contains dynamic group as parent queue. Contributed by Manikandan R 2019-12-05 21:04:42 +01:00
Szilard Nemeth
5cc6f945da YARN-9789. Disable Option for Write Ahead Logs of LogMutation. Contributed by Prabhu Joseph 2019-12-05 20:43:26 +01:00
Szilard Nemeth
4627dd6708 YARN-9780. SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call. Contributed by Prabhu Joseph 2019-12-05 20:38:37 +01:00
Szilard Nemeth
c71befaf8f YARN-9781. SchedConfCli to get current stored scheduler configuration. Contributed by Prabhu Joseph 2019-12-05 20:31:24 +01:00
Szilard Nemeth
520fe2c99b YARN-5106. Provide a builder interface for FairScheduler allocations for use in tests. Contributed by Adam Antal 2019-12-05 17:37:40 +01:00
Sunil G
682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
Wanqiang Ji
c48de9aa2d YARN-9958. Remove the invalid lock in ContainerExecutor (#1704) 2019-12-04 14:35:41 +09:00
Abhishek Modi
a2dadac790 YARN-9990. Testcase fails with Insufficient configured threads: required=16 < max=10. Contributed by Prabhu Joseph. 2019-11-29 11:54:33 +05:30
Szilard Nemeth
ef950b0863 YARN-9290. Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler. Contributed by Prabhu Joseph 2019-11-26 22:04:07 +01:00
Szilard Nemeth
828ab400ee YARN-9362. Code cleanup in TestNMLeveldbStateStoreService. Contributed by Denes Gerencser 2019-11-26 21:45:12 +01:00
prabhujoseph
aa7ab2719f YARN-9991. Fix Application Tag prefix to userid. Contributed by Szilard Nemeth. 2019-11-27 01:59:12 +05:30
Szilard Nemeth
8c9018d5c7 YARN-9899. Migration tool that help to generate CS config based on FS config [Phase 2]. Contributed by Peter Bacsko 2019-11-26 21:22:35 +01:00
Szilard Nemeth
448ffb12ec YARN-9937. addendum: Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2019-11-26 13:41:41 +01:00
Eric Badger
6c20512b68 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger" 
This reverts commit 289bbca870.
 2019-11-25 17:46:42 +00:00
HUAN-PING SU
f8e36e03b4 YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 2019-11-25 15:28:53 +09:00
Szilard Nemeth
4c1a1287bc YARN-9968. Public Localizer is exiting in NodeManager due to NullPointerException. Contributed by Tarun Parimi 2019-11-22 12:58:10 +01:00
Tao Yang
8afabe41e4 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:06:14 +08:00
Szilard Nemeth
dfdc6d6dd9 YARN-9886. Queue mapping based on userid passed through application tag. Contributed by Julia Kinga Marton 2019-11-19 21:14:15 +01:00
Prabhu Joseph
30b2096933 YARN-9986. Allow app owner to signal container via RMWebServices. Contributed by kyungwan nam. 2019-11-19 11:35:00 +00:00
Sunil G
ea68756c0c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:10:41 +05:30
Sunil G
215f2052fc YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:03:02 +05:30
Abhishek Modi
dc3f4fc2f4 YARN-9965. Fix NodeManager failing to start on subsequent times when Hdfs Auxillary Jar is set (addendum). Contributed by Prabhu Joseph. 2019-11-19 08:08:40 +05:30
Eric Badger
0e22e9ab83 YARN-9562. Add Java changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:56:31 +00:00
Eric Badger
289bbca870 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:54:16 +00:00
Szilard Nemeth
27642367ef YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-18 16:29:39 +01:00
Yufei Gu
b83b9ab418 YARN-9537. Add configuration to disable AM preemption. Contributed by ZhouKang 2019-11-12 09:40:16 -08:00
Abhishek Modi
fb512f5087 YARN-9697. Efficient allocation of Opportunistic containers. Contributed by Abhishek Modi. 2019-11-12 16:34:04 +05:30
Szilard Nemeth
30b93f914b YARN-9865. Capacity scheduler: add support for combined %user + %secondary_group mapping. Contributed by Manikandan R 2019-11-11 13:28:02 +01:00
Abhishek Modi
516377bfa6 YARN-9965. Fix NodeManager failing to start when Hdfs Auxillary Jar is set. Contributed by Prabhu Joseph. 2019-11-11 16:01:44 +05:30
Szilard Nemeth
31f172fd96 YARN-9677. Make FpgaDevice and GpuDevice classes more similar to each other. Contributed by kevin su 2019-11-09 14:04:14 +01:00
zhoukang
7d0adddf09 YARN-9689: Support proxy user for Router to support kerberos (#1654) 2019-11-04 17:13:29 -08:00
Sunil G
51e7d1b37e YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 2019-11-04 23:27:49 +05:30
Sunil G
d462308e04 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 2019-11-03 08:47:27 +05:30
Sunil G
9a2e43e29e YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph. 2019-10-31 00:34:27 +05:30
Szilard Nemeth
e3e7daa4f5 YARN-9889. [UI] Add Application Tag column to RM All Applications table. Contributed by Kinga Marton 2019-10-30 16:26:56 +01:00
Abhishek Modi
ed45c13f67 YARN-2442. ResourceManager JMX UI does not give HA State. Contributed by Rohith Sharma and Cyrus Jackson. 2019-10-29 15:54:55 +05:30
Eric Badger
eef34f2d87 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 2019-10-25 17:13:04 +00:00
Eric Yang
ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni. 
Contributed by liusheng
close #1546
 2019-10-24 11:45:57 -04:00
Zhankun Tang
fd84ca5161 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:05:00 +08:00
Eric E Payne
83d148074f YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 2019-10-21 20:30:50 +00:00
Eric Yang
3f7756dc6c YARN-9922. Fixed YARN javadoc errors from YARN-9699. 
Contributed by Peter Bacsko
 2019-10-21 16:16:49 -04:00
Eric Yang
155864da00 YARN-9875. Improve fair scheduler configuration store on HDFS. 
Contributed by Prabhu Joseph
 2019-10-18 20:30:11 -04:00
Eric Yang
72b1bed998 YARN-9884. Make container-executor mount logic modular 
Contributed by Eric Badger
 2019-10-18 19:10:32 -04:00
Mate Szalay-Beko
6d92aa7c30 HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 (#1656). Contributed by Norbert Kalmár, Mate Szalay-Beko 
* HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5

- Add a static initializer for the unit tests using ZooKeeper to enable
the four-letter-words diagnostic telnet commands. (this is an interface
that become disabled by default, so to keep the ZooKeeper 3.4.x behavior
we enabled it for the tests)
- Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt
records. The new ZooKeeper seems to respond quicker during the fail-over
tests than the ZooKeeper, so we made sure to catch all the relevant records
by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().

Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>
 2019-10-18 13:26:20 -07:00
Szilard Nemeth
f0699a7406 YARN-9841. Capacity scheduler: add support for combined %user + %primary_group mapping. Contributed by Manikandan R 2019-10-18 11:19:49 +02:00
Sunil G
3990ffa083 YARN-9909. Offline format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-10-17 06:43:06 +05:30
Eric E Payne
a5034c7988 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 2019-10-16 21:10:08 +00:00
Sunil G
090f73a9aa YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 2019-10-16 18:14:31 +05:30
Szilard Nemeth
a8ef03e961 YARN-9840. Capacity scheduler: add support for Secondary Group rule mapping. Contributed by Manikandan R 2019-10-15 17:22:52 +02:00
Szilard Nemeth
5cc7873a47 YARN-9699. Migration tool that help to generate CS config based on FS config [Phase 1]. Contributed by Peter Bacsko 2019-10-14 17:19:39 +02:00
Szilard Nemeth
ec86f42e40 YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-11 14:01:19 +02:00
Eric Yang
31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN 
Contributed by Prabhu Joseph and Shane Kumpf
 2019-10-10 19:02:02 -04:00
Szilard Nemeth
6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Sunil G
be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Jonathan Hung
5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Sunil G
fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 4510970e2f.
 2019-10-05 09:15:17 +05:30
Sunil G
4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Abhishek Modi
61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
bibinchundatt
425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Sunil G
137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Abhishek Modi
4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Eric Badger
18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Jonathan Hung
606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Szilard Nemeth
bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Jonathan Hung
c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Duo Zhang
bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496) 
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
 2019-09-24 08:50:36 +08:00
Jonathan Hung
3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Sunil G
c474e24c0b YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 2019-09-17 19:41:38 +05:30
Abhishek Modi
da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
HUAN-PING SU
4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
 2019-09-13 10:36:45 -07:00
Abhishek Modi
44850f6784 YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph. 2019-09-12 12:50:10 +05:30