big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,204 Commits 2 Branches 2 Tags 264 MiB
d7d36b9d2a
Commit Graph

2558 Commits

Author SHA1 Message Date
slfan1989
4a23389f5e
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-03-10 15:16:28 +00:00
Varun Saxena
157af0cb22 YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary

(cherry picked from commit 2a0dc2ab2f)
 2023-03-05 23:02:17 +05:30
Chris Nauroth
1f270d8a5e YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
(cherry picked from commit 6b67373d10)
 2022-12-28 19:02:02 +00:00
curie71
290dc7817c YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 9668a85d40)
 2022-12-28 00:00:18 +00:00
Bence Kosztolnik
7190fcf713 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:20:35 +01:00
susheel-gupta
f9fac84f43 YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:26:05 +01:00
K0K0V0K
8b748c1cb8 YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> (#5190) 
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit ee7d1787cd)
 2022-12-08 18:15:08 +00:00
Chris Nauroth
33293d4ba4 YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 
(cherry picked from commit bfb84cd7f6)
 2022-10-28 18:13:57 +00:00
Steve Loughran
19f8e4f34d
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-21 14:16:22 +01:00
Ashutosh Gupta
3b3bd89084 YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING (#4721) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f02ff1afe2)
 2022-08-16 19:07:42 +09:00
Szilard Nemeth
ebe3ed5cd3 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 
(cherry picked from commit 7cb887e6c2)
 2022-07-07 21:04:15 +00:00
Benjamin Teke
838b63d836 YARN-10974. Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu. 2022-06-22 18:20:09 +02:00
zhengchenyu
d7de378b22
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. (#4408) 
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.

Contributed by zhengchenyu
 2022-06-17 19:51:56 +01:00
Ashutosh Gupta
e0732baeb8 YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e3e9369c1d)
 2022-05-26 15:58:47 +09:00
Owen O'Malley
1f111d6a41 YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 
(cherry picked from commit f390edaec4)
 2022-05-24 05:17:34 +00:00
Szilard Nemeth
90ec4418c7 YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-18 14:34:08 +02:00
Szilard Nemeth
4f112e3138 YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-18 14:25:35 +02:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
Fei Hui
5a38ed2f22 HADOOP-17276. Extend CallerContext to make it include many items (#2327) 
Cherry-picked from d0d10f7e by Owen O'Malley
 2022-03-14 10:28:38 -07:00
Szilard Nemeth
192f53283b YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-03-02 16:04:38 +01:00
Szilard Nemeth
3ef3c5a05b YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok 2022-03-02 14:45:31 +01:00
Szilard Nemeth
f06f44b1c2 YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2022-03-02 14:23:00 +01:00
Szilard Nemeth
935619a28c YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-03-01 18:05:04 +01:00
Ashutosh Gupta
1c99810b89 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)
 2021-12-23 17:16:09 +09:00
Eric Payne
ccaba2561a YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075d)
 2021-12-21 19:20:21 +00:00
Shubham Gupta
c44f109860
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2021-11-19 00:23:48 +09:00
Ahmed Hussein
742d88b1c6 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-21 17:04:29 +00:00
Benjamin Teke
700045896c
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3504) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-10-12 18:05:50 +02:00
Neil
88deac0479
YARN-10970. Standby RM should expose prom endpoint (#3480) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4bd0c36189)
 2021-09-29 15:48:02 +09:00
Eric Badger
52ba50fd3c YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 
(cherry picked from commit 43f0a34dd4)
 2021-09-16 16:46:44 +00:00
Szilard Nemeth
6c68211062 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-14 18:08:34 +02:00
Szilard Nemeth
0a726250ea
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f7)
 2021-09-01 10:44:15 +09:00
Szilard Nemeth
a272adc5fa YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-07-29 19:21:58 +02:00
Szilard Nemeth
72801be13a YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-07-28 14:55:19 +02:00
zhuqi-lucas
c31618e6b9 YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-22 10:12:32 +08:00
Jim Brennan
b3481062e0 YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 
(cherry picked from commit 632f64cadb)
 2021-07-15 14:26:03 +00:00
Jim Brennan
47b3939009 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 
(cherry picked from commit dc6f456e95)
 2021-06-28 14:55:26 +00:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Jim Brennan
53a1c7653f YARN-10337. Amendment to fix import as in HADOOP-17100 2021-05-19 22:00:55 +00:00
Prabhu Joseph
1b3e4cf9ce YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.

(cherry picked from commit 2bbd00dff4)
 2021-05-19 21:19:05 +00:00
zhuqi-lucas
7d2eeaecc8 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-05-19 21:24:26 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
lujiefsi
137e20cc9b
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 13:53:27 +09:00
Peter Bacsko
051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Szilard Nemeth
3d715c2e4c
YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 
(cherry picked from commit eacbe07b56)
 2021-05-07 14:32:17 +09:00
Szilard Nemeth
3303aa5947
YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 
(cherry picked from commit 8620984b8d)
 2021-05-07 14:26:58 +09:00
Wei-Chiu Chuang
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029)
 2021-05-02 11:12:37 +09:00
Eric Badger
003deeeecf YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 22:14:57 +00:00
Eric Badger
1960924d07 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7)
 2021-04-20 17:46:05 +00:00
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 17:51:01 +00:00