big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,777 Commits 2 Branches 2 Tags 264 MiB
ff6ec20d84
Commit Graph

4286 Commits

Author SHA1 Message Date
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt
61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Neil
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Eric Badger
4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.
 2020-12-21 23:10:31 +00:00
zhuqi
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. (#2563) 
Qi Zhu via Zhankun Tang
 2020-12-21 10:04:16 +08:00
Akira Ajisaka
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka 2020-12-16 17:26:15 +01:00
Szilard Nemeth
df35c7f519 YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak 2020-12-15 17:30:14 +01:00
adamantal
3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
adamantal
0a45bd034e YARN-10520. Deprecated the residual nested class for the LCEResourceHandler (#2527). Contributed by Wanqiang Ji 2020-12-09 12:55:24 +01:00
zhuqi
d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler (#2494) 
Contributed by Qi Zhu.
 2020-12-09 19:48:39 +08:00
Eric Badger
7d3c8ef606 YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.
 2020-12-08 00:53:24 +00:00
adamantal
9969745343 YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy 2020-12-03 09:55:06 +01:00
Eric Payne
fa773a8326 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-01 22:06:47 +00:00
Ankit Kumar
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-25 13:00:09 +09:00
Ahmed Hussein
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions (#2461) 2020-11-24 11:09:30 -08:00
Szilard Nemeth
0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth
5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Peter Bacsko
567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
Eric E Payne
55339c2bdd YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 2020-11-05 21:50:46 +00:00
Eric E Payne
0461a07c01 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 16:55:06 +00:00
Surendra Singh Lilhore
7169ec4509
YARN-10442. RM should make sure node label file highly available. (#2390) 
* YARN-10442. RM should make sure node label file highly available.

* YARN-10442. Corrected method name.
            Added license header for file TestNodeLabelFileReplication.java
            Added Test code.

* YARN-10442. Changed property name and log.

* YARN-10442. Changed default replication for FS store file.
 2020-10-30 23:05:16 +05:30
Wangda Tan
c47c9fd65d YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. (Peter Bacsko via wangda) 
Change-Id: I518dc925187ce55e9d35a37ba20878c0f4e37e5c
 2020-10-30 09:49:48 -07:00
Jim Brennan
8ee6bc2518 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 19:49:18 +00:00
Jonathan Hung
bab5bf9743 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:33:18 -07:00
Sunil G
7435604a91 YARN-10453. Add partition resource info to get-node-labels and label-mappings api responses. Contributed Akhil PB. 2020-10-22 12:52:58 +05:30
Szilard Nemeth
31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Eric Badger
8b8c672780 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-15 17:57:43 +00:00
Szilard Nemeth
42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Jim Brennan
deb35a32ba YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:46:47 +00:00
Jim Brennan
b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Adam Antal
a1f7e760df YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-05 10:09:14 +02:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Adam Antal
bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
Szilard Nemeth
a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Shubham Gupta
e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
Szilard Nemeth
eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Jim Brennan
90894ea641 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-14 21:19:14 +00:00
Eric E Payne
9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00
Szilard Nemeth
773ac799c6 YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko 2020-09-09 15:12:58 +02:00
Brahma Reddy Battula
43572fc7f8 YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 2020-09-09 17:08:13 +05:30
Peter Bacsko
aba4a506d6 YARN-10424. Adapt existing AppName and UserGroupMapping unittests to ensure backwards compatibility. Contributed by Benjamin Teke. 2020-09-09 09:41:11 +02:00
Peter Bacsko
c4fb4044b2 YARN-10415. Create a group matcher which checks ALL groups of the user. Contributed by Gergely Pollak. 2020-09-08 10:57:00 +02:00
Adam Antal
34fe74da0e YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 2020-09-07 11:44:09 +02:00
Peter Bacsko
360bbcd3bc YARN-10411. Create an allowCreate flag for MappingRuleAction. Contributed by Gergely Pollak. 2020-09-07 11:39:03 +02:00
Peter Bacsko
1841a5bb03 YARN-10376. Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak. 2020-09-05 10:00:33 +02:00
Szilard Nemeth
696e4fe50e Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja 2020-09-04 09:59:54 +02:00
Peter Bacsko
5c15815773 YARN-10375. CS Mapping rule config parser should return MappingRule objects. Contributed by Gergely Pollak. 2020-09-03 18:54:22 +02:00
Peter Bacsko
288dab7563 YARN-10372. Create MappingRule class to represent each CS mapping rule. Contributed by Gergely Pollak 2020-09-03 15:02:48 +02:00
Masatake Iwasaki
5e12dc51f0
YARN-10419. Javadoc error in hadoop-yarn-server-common module. (#2271) 2020-09-03 16:17:52 +09:00
Peter Bacsko
c7dab2b23e YARN-10374. Create Actions for CS mapping rules. Contributed by Gergely Pollak. 2020-09-01 17:38:49 +02:00
Adam Antal
73a0d149e2 YARN-10386. Addendum patch: Fixing ASF Warnings. Contributed by Peter Bacsko. 2020-09-01 14:39:24 +02:00
Szilard Nemeth
4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Szilard Nemeth
56a5c360a1 YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak 2020-08-29 21:34:55 +02:00
Szilard Nemeth
f4f872b778 YARN-10371. Create variable context class for CS queue mapping rules. Contributed by Gergely Pollak 2020-08-29 17:31:48 +02:00
Szilard Nemeth
a888d580d8 YARN-10409. Improve MockQueueHierarchyBuilder to detect queue ambiguity. Contributed by Gergely Pollak 2020-08-29 15:24:50 +02:00
Szilard Nemeth
44542863f4 YARN-10408. Extract MockQueueHierarchyBuilder to a separate class. Contributed by Gergely Pollak 2020-08-28 17:35:45 +02:00