slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. ( #4954 )
2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. ( #4946 )
2022-10-11 13:30:48 -07:00
Szilard Nemeth
0c515b0ef0
YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal
2022-10-10 15:30:33 +02:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 ( #4937 )
...
Contributed by PJ Fanning
2022-10-07 15:44:01 +01:00
slfan1989
342c4856b8
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. ( #4852 )
2022-09-19 13:14:55 -07:00
9uapaw
5b85af87f0
YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta.
2022-09-09 14:38:41 +02:00
slfan1989
37e213c3fc
YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation. ( #4764 )
2022-09-01 16:35:20 -07:00
slfan1989
4031b0774e
YARN-11253. Add Configuration to delegationToken RemoverScanInterval. ( #4751 )
2022-08-27 10:02:59 -07:00
slfan1989
f8b9dd911c
YARN-11219. [Federation] Add getAppActivities, getAppStatistics REST APIs for Router. ( #4757 )
2022-08-26 16:01:17 -07:00
Ashutosh Gupta
c294a414b9
YARN-9425. Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService ( #4731 ). Contributed by groot and Shen Yinjie.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-08-22 03:40:00 +05:30
slfan1989
f75c58a1ca
YARN-11252. Yarn Federation Router Supports Update / Delete Reservation in MemoryStore. ( #4741 )
2022-08-18 21:13:43 -07:00
Ashutosh Gupta
f02ff1afe2
YARN-11248. Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING ( #4721 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-08-16 19:07:05 +09:00
slfan1989
d383cc4525
YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. ( #4711 )
2022-08-13 10:37:20 -07:00
slfan1989
133e8aabf0
YARN-11227. [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router. ( #4715 )
2022-08-10 14:53:46 -07:00
slfan1989
c5eba323bc
YARN-6972. Adding RM ClusterId in AppInfo. ( #4673 )
2022-08-03 09:35:40 -07:00
slfan1989
1f0a71a92b
YARN-8973. [Router] Add missing methods in RMWebProtocol. ( #4664 )
2022-08-02 14:07:09 -07:00
slfan1989
57da4bb0a1
YARN-11220. [Federation] Add getLabelsToNodes, getClusterNodeLabels, getLabelsOnNode REST APIs for Router ( #4657 )
2022-08-02 12:09:55 -07:00
slfan1989
e994635a95
YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router. ( #4614 )
2022-07-28 11:53:04 -07:00
9uapaw
bf570bd4ac
YARN-11063. Support auto queue creation template wildcards for arbitrary queue depths. Contributed by Bence Kosztolnik.
2022-07-28 17:32:20 +02:00
Szilard Nemeth
f4b635c4dc
YARN-11211. QueueMetrics leaks Configuration objects when validation API is called multiple times. Contributed by Andras Gyori
2022-07-21 14:20:34 +02:00
PJ Fanning
34e548cb62
HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api ( #4547 )
...
This downgrades jackson from the version switched to in
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.
The 2.12.7 release still contains the fix for CVE-2020-36518.
Contributed by PJ Fanning
2022-07-17 21:37:54 +05:30
Ashutosh Gupta
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping ( #4515 )
...
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-07-02 21:58:56 +05:30
Tamas Domok
3cad632709
YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok.
...
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
2022-07-01 10:50:48 +02:00
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' ( #4422 ). Contributed by fanshilun.
2022-06-29 15:20:41 +05:30
Ashutosh Gupta
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation ( #4487 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-23 21:59:27 +05:30
Szilard Nemeth
3a66348fda
YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori
2022-06-20 15:03:58 +02:00
zhengchenyu
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. ( #4408 )
...
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.
Contributed by zhengchenyu
2022-06-17 19:49:36 +01:00
Benjamin Teke
020201cb65
Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu.
2022-06-17 19:28:32 +02:00
Ashutosh Gupta
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project ( #4381 ). Contributed by Ashutosh Gupta.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-05-31 23:17:11 +05:30
Ashutosh Gupta
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* ( #4271 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-26 15:51:55 +09:00
Owen O'Malley
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. ( #4350 )
2022-05-24 14:07:19 +09:00
9uapaw
0e6a6d1880
YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation
2022-05-18 18:22:42 +02:00
9uapaw
54cd0174c0
YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe
2022-05-18 18:16:34 +02:00
Jian Chen
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases ( #4110 )
...
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:11:42 +09:00
Szilard Nemeth
eced5bec68
YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth
2022-05-13 13:16:12 +02:00
Szilard Nemeth
5e2f4339fa
YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok
2022-05-12 13:42:06 +02:00
Szilard Nemeth
9af3eabdca
YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth
2022-05-11 18:01:31 +02:00
Szilard Nemeth
b62d6ce6fd
YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori
2022-05-11 14:29:05 +02:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 ( #3988 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-28 14:18:19 +09:00
章锡平
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. ( #4175 )
2022-04-15 17:50:49 +09:00
章锡平
3b46aae977
YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1
2022-04-13 15:57:18 +02:00
Junfan Zhang
e044a46f97
YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang
2022-03-31 12:39:56 +02:00
PJ Fanning
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 ( #3990 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-26 20:31:16 +09:00
Tamas Domok
da09d68056
YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok
2022-03-25 21:53:18 +01:00
9uapaw
921267ca31
YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang.
2022-03-23 08:57:00 +01:00
9uapaw
2beb7296fb
YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang.
2022-03-22 13:14:43 +01:00
9uapaw
c3124a3f6e
YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke.
2022-03-21 14:55:49 +01:00
Szilard Nemeth
e2701e227f
YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang.
2022-03-21 14:37:10 +01:00
Szilard Nemeth
4537b34e1c
YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang.
2022-03-21 14:18:27 +01:00
Szilard Nemeth
ed65aa2324
YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori
2022-03-10 22:22:58 +01:00
Szilard Nemeth
481da19494
YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke
2022-03-10 22:15:35 +01:00
9uapaw
383b73417d
YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-03-10 13:11:19 +01:00
9uapaw
9539ff108a
YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth
2022-03-10 08:40:43 +01:00
Szilard Nemeth
db8ae4b654
YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori
2022-03-09 19:36:09 +01:00
Szilard Nemeth
2ece95064b
YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori
2022-03-09 19:29:53 +01:00
Viraj Jasani
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing ( #4027 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-08 21:32:51 +09:00
9uapaw
851a4dc2d9
YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth
2022-03-07 09:01:18 +01:00
Szilard Nemeth
379baa5eb6
YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori
2022-03-03 16:44:12 +01:00
Szilard Nemeth
356d337d1e
YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok
2022-03-02 16:19:17 +01:00
9uapaw
9e475aede6
YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke
2022-03-02 11:16:24 +01:00
Szilard Nemeth
365375412f
YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori
2022-02-22 12:26:35 +01:00
Szilard Nemeth
0463498adc
YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori
2022-02-22 11:55:34 +01:00
Szilard Nemeth
7dd288ce8c
YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori
2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897
YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke
2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba
YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke
2022-02-17 19:43:37 +01:00
Szilard Nemeth
48e02f7998
YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok
2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c
YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5
YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok
2022-02-17 19:03:44 +01:00
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… ( #3195 )
2022-02-01 09:37:40 -08:00
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… ( #3741 )
...
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
minni31
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse ( #3766 )
2022-01-25 09:25:18 -08:00
Andras Gyori
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable ( #3859 )
2022-01-11 14:38:59 +08:00
Ashutosh Gupta
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch ( #3793 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
Eric Payne
e2d6fd075d
YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).
2021-12-21 19:05:39 +00:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
Szilard Nemeth
07141426e0
YARN-11050 ( #3805 )
2021-12-18 11:52:03 +08:00
Szilard Nemeth
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class ( #3800 ). Contributed by Szilard Nemeth
2021-12-17 00:18:14 +01:00
Szilard Nemeth
aec9cdb467
YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok
2021-12-16 23:39:18 +01:00
Szilard Nemeth
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API ( #3799 ). Contributed by Szilard Nemeth
2021-12-16 15:53:08 +01:00
Szilard Nemeth
6ea1685abd
YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke
2021-12-16 00:01:09 +01:00
Szilard Nemeth
843f66f4dc
Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke
2021-12-14 22:00:46 +01:00
Szilard Nemeth
364d38f00d
YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke
2021-12-14 14:57:08 +01:00
Szilard Nemeth
a5bcf4c792
YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke
2021-12-13 21:57:46 +01:00
Szilard Nemeth
898055e204
YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke
2021-12-13 16:12:55 +01:00
Szilard Nemeth
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
Szilard Nemeth
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
Andrew Chung
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
Szilard Nemeth
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
Szilard Nemeth
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
Viraj Jasani
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
Szilard Nemeth
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
Szilard Nemeth
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
Szilard Nemeth
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
Szilard Nemeth
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
Andras Gyori
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
Szilard Nemeth
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00