big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,886 Commits 2 Branches 2 Tags 264 MiB
0780710f25
Commit Graph

4750 Commits

Author SHA1 Message Date
Ashutosh Gupta
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping (#4515) 
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-07-02 21:58:56 +05:30
Tamas Domok
3cad632709 YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok. 
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
 2022-07-01 10:50:48 +02:00
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Ashutosh Gupta
a177232ebc
YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down (#4492) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-28 09:32:07 +05:30
Ashutosh Gupta
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation (#4487) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-23 21:59:27 +05:30
slfan1989
0af4bb3b42
YARN-11192. TestRouterWebServicesREST failing after YARN-9827. (#4484). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-23 13:21:36 +05:30
Szilard Nemeth
3a66348fda YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori 2022-06-20 15:03:58 +02:00
slfan1989
62e4476102
YARN-10122. Support signalToContainer API for Federation. (#4421) 2022-06-17 16:38:36 -07:00
zhengchenyu
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. (#4408) 
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.

Contributed by zhengchenyu
 2022-06-17 19:49:36 +01:00
Benjamin Teke
020201cb65 Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu. 2022-06-17 19:28:32 +02:00
slfan1989
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation (#4396) 2022-06-08 17:34:43 -07:00
slfan1989
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation (#4341) 2022-06-02 12:54:29 -07:00
Ashutosh Gupta
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project (#4381). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 23:17:11 +05:30
Ashutosh Gupta
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-26 15:51:55 +09:00
slfan1989
678768035d
YARN-11137. Improve log message in FederationClientInterceptor (#4336) 2022-05-24 17:21:04 -07:00
Owen O'Malley
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 2022-05-24 14:07:19 +09:00
slfan1989
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation (#4317) 2022-05-20 13:19:42 -07:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
9uapaw
0e6a6d1880 YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation 2022-05-18 18:22:42 +02:00
9uapaw
54cd0174c0 YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe 2022-05-18 18:16:34 +02:00
slfan1989
6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274) 2022-05-15 09:16:06 -07:00
Ashutosh Gupta
6985f9aabe
YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:41:06 +09:00
Jian Chen
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110) 
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:11:42 +09:00
Szilard Nemeth
eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Szilard Nemeth
5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
Szilard Nemeth
9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth
b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989
09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
PJ Fanning
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
章锡平
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. (#4175) 2022-04-15 17:50:49 +09:00
章锡平
3b46aae977 YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1 2022-04-13 15:57:18 +02:00
Junfan Zhang
e044a46f97 YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang 2022-03-31 12:39:56 +02:00
PJ Fanning
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-26 20:31:16 +09:00
Tamas Domok
da09d68056 YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok 2022-03-25 21:53:18 +01:00
9uapaw
921267ca31 YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang. 2022-03-23 08:57:00 +01:00
9uapaw
2beb7296fb YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang. 2022-03-22 13:14:43 +01:00
9uapaw
c3124a3f6e YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke. 2022-03-21 14:55:49 +01:00
Szilard Nemeth
e2701e227f YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang. 2022-03-21 14:37:10 +01:00
Szilard Nemeth
4537b34e1c YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang. 2022-03-21 14:18:27 +01:00
Szilard Nemeth
ed65aa2324 YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori 2022-03-10 22:22:58 +01:00
Szilard Nemeth
481da19494 YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke 2022-03-10 22:15:35 +01:00
9uapaw
383b73417d YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-03-10 13:11:19 +01:00
9uapaw
9539ff108a YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth 2022-03-10 08:40:43 +01:00
Szilard Nemeth
db8ae4b654 YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori 2022-03-09 19:36:09 +01:00
Szilard Nemeth
2ece95064b YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori 2022-03-09 19:29:53 +01:00
Viraj Jasani
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing (#4027) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-08 21:32:51 +09:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
9uapaw
851a4dc2d9 YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth 2022-03-07 09:01:18 +01:00
Szilard Nemeth
379baa5eb6 YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-03-03 16:44:12 +01:00
Szilard Nemeth
356d337d1e YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok 2022-03-02 16:19:17 +01:00
9uapaw
9e475aede6 YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke 2022-03-02 11:16:24 +01:00
Szilard Nemeth
365375412f YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori 2022-02-22 12:26:35 +01:00
Szilard Nemeth
0463498adc YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori 2022-02-22 11:55:34 +01:00
Szilard Nemeth
7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth
48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741) 
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-02-01 12:18:54 +05:30
minni31
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse (#3766) 2022-01-25 09:25:18 -08:00
Andrew Chung
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container (#3779) 2022-01-24 08:03:36 -08:00
Gera Shegalov
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-17 16:20:48 +09:00
Andras Gyori
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
minni31
3bf014d871
YARN-8859. Add audit logs for router service (#3607) 
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-01-03 10:38:59 +05:30
Cheng Pan
4db27acf06
YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 11:08:04 +09:00
Ashutosh Gupta
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 17:14:51 +09:00
Eric Payne
e2d6fd075d YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:05:39 +00:00
Viraj Jasani
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled (#3802) 2021-12-20 18:02:57 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Szilard Nemeth
07141426e0
YARN-11050 (#3805) 2021-12-18 11:52:03 +08:00
Szilard Nemeth
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class (#3800). Contributed by Szilard Nemeth 2021-12-17 00:18:14 +01:00
Szilard Nemeth
aec9cdb467 YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok 2021-12-16 23:39:18 +01:00
Szilard Nemeth
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API (#3799). Contributed by Szilard Nemeth 2021-12-16 15:53:08 +01:00
Szilard Nemeth
6ea1685abd YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke 2021-12-16 00:01:09 +01:00
Viraj Jasani
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (#3796) 2021-12-15 16:47:51 +08:00
Szilard Nemeth
843f66f4dc Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke 2021-12-14 22:00:46 +01:00
Szilard Nemeth
364d38f00d YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke 2021-12-14 14:57:08 +01:00
Szilard Nemeth
a5bcf4c792 YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke 2021-12-13 21:57:46 +01:00
Szilard Nemeth
898055e204 YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke 2021-12-13 16:12:55 +01:00
Viraj Jasani
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown (#3768) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-13 14:03:32 +09:00
Szilard Nemeth
f544ef4a51 YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok 2021-12-10 15:09:53 +01:00
Szilard Nemeth
a0d8cde133 YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs 2021-12-09 17:51:44 +01:00
Andrew Chung
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy (#3717) 2021-12-08 10:02:10 -08:00
Szilard Nemeth
66a08e933a YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok 2021-12-08 16:41:35 +01:00
Szilard Nemeth
19430118c6 YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2021-12-08 16:03:42 +01:00
Viraj Jasani
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3749) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 16:52:22 +09:00
Szilard Nemeth
126079612c YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori 2021-12-07 15:55:16 +01:00
Szilard Nemeth
7284d23476 YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke 2021-12-07 15:43:55 +01:00
Szilard Nemeth
60440a836a YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok 2021-12-07 15:26:07 +01:00
Szilard Nemeth
d75f2db2f9 YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2021-12-07 08:42:57 +01:00
Ashutosh Gupta
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-06 19:45:44 +09:00
Andras Gyori
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars (#3710) 
Contributed by Andras Gyori
 2021-12-03 16:44:58 +00:00
Viraj Jasani
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile (#3722) 2021-12-02 10:40:30 +08:00
Sushmasree-28
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files (#3735) 
Contributed by  sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
 2021-11-30 00:00:25 +05:30
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 (#3642) 2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources (#3646) 2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules (#3687) 2021-11-23 13:36:22 +09:00
Szilard Nemeth
7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
Szilard Nemeth
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration (#3570). Contributed by Szilard Nemeth 2021-10-28 15:19:32 +02:00
Szilard Nemeth
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue (#3551) contributed by Szilard Nemeth 2021-10-27 19:03:45 +02:00
Szilard Nemeth
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler (#3581) Contributed by Szilard Nemeth 2021-10-27 17:13:49 +02:00
Jack
9cfd8d0a83 YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth 2021-10-23 14:47:09 +02:00
9uapaw
32ecaed9c3 YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori 2021-10-22 17:32:33 +02:00
Adam Antal
23772d946b YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal 2021-10-22 16:33:03 +02:00
Ahmed Hussein
d286994009 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-20 22:18:36 +00:00
Szilard Nemeth
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth 2021-10-20 15:52:37 +02:00
Andras Gyori
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori 2021-10-20 12:56:41 +02:00
Szilard Nemeth
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule (#3560) 
* YARN-10958. Initial commit

* Fix javadoc + behaviour

* Fix review comments

* fix checkstyle + blanks

* fix checkstyle + blanks

* Fix checkstyle + blanks
 2021-10-20 10:48:42 +02:00
9uapaw
616cea2e80 YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori 2021-10-19 13:06:45 +02:00
Szilard Nemeth
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth 2021-10-19 12:24:58 +02:00
lujiefsi
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk (#3552) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 15:24:15 +09:00
Viraj Jasani
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules (#3541) 2021-10-14 18:03:01 +09:00
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
Szilard Nemeth
8c271caa81 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-10-12 17:35:06 +02:00
Benjamin Teke
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-10-07 20:11:42 +02:00
9uapaw
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori 2021-10-07 17:09:38 +02:00
Tamas Domok
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-07 16:21:05 +02:00
9uapaw
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori 2021-10-01 04:20:36 +02:00
Tamas Domok
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-01 04:05:52 +02:00
Neil
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint (#3480) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-29 15:46:55 +09:00
Akshat Bordia
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics (#3439) 2021-09-28 09:51:18 -07:00
Tamas Domok
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-21 16:13:04 +02:00
9uapaw
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Adam Antal
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
Szilard Nemeth
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
Eric Badger
43f0a34dd4 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-15 20:03:45 +00:00
Benjamin Teke
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources (#3418) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-14 18:05:40 +02:00
Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy (#3403) 
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic (#3390) 
- capacityConfigType update is extracted to a separate method
 - validation logic is extracted to a helper function
 - min resource must not be greater than max resource is now checked
   after the max resource is updated

Change-Id: I731c2639281721afed32c30854bafcf048d6ee28

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:30:44 +02:00
Weihao Zheng
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor (#3389) 
Contributed by Weihao Zheng
 2021-09-13 11:06:53 +08:00
Jack
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF (#3352) 
Contributed by Jie Wang <jie.wang@hulu.com>
 2021-09-13 10:54:11 +08:00
Benjamin Teke
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate (#3396) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 17:32:42 +02:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue (#3392) 2021-09-10 16:59:46 +02:00
Benjamin Teke
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic (#3407) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 16:48:58 +02:00
Tamas Domok
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue (#3402) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-10 15:57:46 +02:00
Akshat Bordia
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor (#3325) 2021-09-09 09:20:57 -07:00
Akshat Bordia
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report (#3252) 2021-09-08 10:36:56 -07:00
Szilard Nemeth
2ff3fc50e4 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-08 18:01:39 +02:00
Jack
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field (#3382) 
Co-authored-by: Jie Wang <jie.wang@hulu.com>
 2021-09-08 16:19:29 +02:00
Benjamin Teke
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario (#3339) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-08 16:11:04 +02:00
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files (#3318) 
* YARN 10884 : Parse log files which has empty owner

* Removing the whitespace

* Added a test case for null user

* Fixed indentations

* Fixed the indentation for test cases

Co-authored-by: Swathi C <cswathi@microsoft.com>
 2021-09-07 21:37:51 +05:30
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-27 23:09:54 +02:00
Szilard Nemeth
e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287) 
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
 2021-08-17 14:18:55 +05:30
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
Szilard Nemeth
fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203) 
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135) 
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
 2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 (#3204) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-16 12:17:17 +09:00
Jim Brennan
632f64cadb YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 14:23:31 +00:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
Jim Brennan
dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Peter Bacsko
0934e783cf YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori. 2021-06-24 20:15:10 +02:00
Szilard Nemeth
6562391737 YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-06-16 18:55:09 +02:00
Szilard Nemeth
428478bbe2 YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori 2021-06-16 18:26:58 +02:00
Szilard Nemeth
e31d06032b YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke 2021-06-14 22:33:04 +02:00
Szilard Nemeth
7003997e36 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-06-12 14:49:52 +02:00
Viraj Jasani
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project (#3075) 2021-06-09 15:15:47 +09:00
Viraj Jasani
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-08 19:37:23 -07:00
Prabhu Josephraj
9445abb500 YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar 2021-06-08 20:37:40 +05:30
zhuqi-lucas
ec16b1d3b9 YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke. 2021-06-08 21:03:43 +08:00
Hideyuki Furue
9c7b8cf54e
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-06 07:05:29 +09:00
Szilard Nemeth
200eec8f2e YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko 2021-06-03 13:27:36 +02:00
Szilard Nemeth
2707f69251 YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak 2021-06-01 16:01:39 +02:00
Gergely Pollak
e9339aa376 YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth 2021-06-01 15:57:22 +02:00
Szilard Nemeth
b86a6eb871 YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori 2021-05-25 18:27:53 +02:00
Szilard Nemeth
2541efa496 YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori 2021-05-25 13:55:59 +02:00
Viraj Jasani
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project (#3033) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 18:10:20 +09:00
zhuqi-lucas
59172ada90 YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju. 2021-05-24 23:12:07 +08:00
Szilard Nemeth
1e44bdb84c YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke 2021-05-22 14:55:01 +02:00
Peter Bacsko
eb72628e15 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-17 21:07:29 +02:00
Peter Bacsko
8891e5c028 YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju. 2021-05-17 13:30:12 +02:00
lujiefsi
d92a25b790
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-17 13:44:38 +09:00
zhuqi
e7f0e8073b YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu. 2021-05-14 13:36:07 +08:00
zhuqi
d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. (#2911) 
Contributed by Qi Zhu.
 2021-05-14 13:12:28 +08:00
Peter Bacsko
626be24c3e YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori. 2021-05-12 14:54:47 +02:00
Akira Ajisaka
7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-06 23:37:45 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Eric Badger
803ac4b1a0 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 2021-04-29 17:42:13 +00:00
Peter Bacsko
9166bfeb74 YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu. 2021-04-24 10:10:10 +02:00
Eric Badger
6857a05d6a YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
 2021-04-23 22:02:04 +00:00
Jim Brennan
1cbe35946b YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 2021-04-23 21:35:09 +00:00
Peter Bacsko
14a84c47b0 YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja. 2021-04-23 17:38:47 +02:00
Szilard Nemeth
3b6b21e459 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 2021-04-23 16:17:29 +02:00
Szilard Nemeth
919daec36b YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak 2021-04-23 16:12:12 +02:00
Szilard Nemeth
f76a2a7606 YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko 2021-04-23 16:07:58 +02:00
Eric Badger
6cb90005a7 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 2021-04-20 17:34:49 +00:00
Viraj Jasani
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-17 20:47:44 +09:00
Eric Badger
213d3deb26 YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 00:34:15 +00:00
Peter Bacsko
ca9aa91d10 YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori. 2021-04-08 12:42:48 +02:00
Szilard Nemeth
9cd69c20c4 YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori 2021-04-07 11:52:21 +02:00
Eric Badger
26b8f678b2 YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 01:16:14 +00:00
Peter Bacsko
158758c5bf YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu. 2021-04-01 16:09:52 +02:00
Peter Bacsko
9f1655baf2 YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu. 2021-04-01 11:39:40 +02:00
Peter Bacsko
a0deda1a77 YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 2021-04-01 09:21:15 +02:00
Szilard Nemeth
6fd0c661b6 YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak 2021-03-31 16:14:21 +02:00
Peter Bacsko
ff6ec20d84 YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu. 2021-03-31 10:55:14 +02:00
Akira Ajisaka
8ad77a26c9
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 2021-03-30 17:46:51 +09:00
Eric Badger
19e418c10d YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 2021-03-25 22:33:58 +00:00
Peter Bacsko
ceb75e1e2a YARN-10674. fs2cs should generate auto-created queue deletion properties. Contributed by Qi Zhu. 2021-03-24 08:15:06 +01:00
Jim Brennan
174f3a96b1 YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:21:45 +00:00
Jack Jiang
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Cyrus Jackson
cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications (#2674). Contributed by Cyrus Jackson 2021-03-19 15:49:05 +05:30
Eric Badger
3d4ea4b336 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-03-18 19:36:50 +00:00
Eric Badger
181b9ddc51 YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 2021-03-18 17:45:09 +00:00
Peter Bacsko
ce6bfd5718 YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu. 2021-03-18 13:40:16 +01:00
Peter Bacsko
38495af325 YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 2021-03-18 12:46:11 +01:00
Szilard Nemeth
a5745711dd YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak 2021-03-18 12:43:01 +01:00
Peter Bacsko
d7eeca4d0c YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu. 2021-03-18 11:49:16 +01:00
Eric Badger
49f89f1d3d YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 2021-03-17 18:11:37 +00:00
Peter Bacsko
3e58d5611d YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu. 2021-03-17 13:38:20 +01:00
Wilfred Spiegelenburg
f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it 
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.

Contributed by: Siddharth Ahuja
 2021-03-17 10:55:05 +11:00
Peter Bacsko
73394fabc7 YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu. 2021-03-16 16:09:12 +01:00
Peter Bacsko
b80588b688 YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu. 2021-03-16 15:23:27 +01:00
zhuqi
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. (#2760) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-16 13:50:29 +09:00
Eric Payne
aa4c17b9d7 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 2021-03-15 19:09:40 +00:00
Jim Brennan
5f067cf0f3 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 2021-03-12 17:17:31 +00:00
Szilard Nemeth
5db4c0bf70 YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak 2021-03-12 14:10:16 +01:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Brahma Reddy Battula
b2a565629d YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 2021-03-09 20:26:07 +05:30
Peter Bacsko
d5e035dbe1 YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-03-09 14:33:14 +01:00
Peter Bacsko
3851994cd6 Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu." 
This reverts commit 369f75b7a7.
 2021-03-09 14:32:02 +01:00
Peter Bacsko
369f75b7a7 YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu. 2021-03-09 14:28:23 +01:00
Peter Bacsko
20416bc62d YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth. 2021-03-09 11:34:37 +01:00
Peter Bacsko
e472ee2aa5 YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth. 2021-03-09 11:28:24 +01:00
Peter Bacsko
04cd3115ba YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu. 2021-03-08 12:29:44 +01:00
Peter Bacsko
c6b30a59dd YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth. 2021-03-05 23:13:35 +01:00
Eric Badger
9d088639bf YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.
 2021-03-05 19:46:40 +00:00
Peter Bacsko
e82e7c597a YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu. 2021-03-05 15:56:56 +01:00
Peter Bacsko
e19c00925f YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu. 2021-03-05 13:18:06 +01:00
Peter Bacsko
6699198b54 YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. 2021-03-04 17:18:35 +01:00
Peter Bacsko
a85aeee876 YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. 2021-03-04 12:23:11 +01:00
Neil
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 2021-03-04 14:52:58 +05:30
Peter Bacsko
7e8040e6ad YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. 2021-03-03 17:44:30 +01:00
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
Peter Bacsko
947b50489d YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke. 2021-02-26 21:57:54 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Jonathan Hung
ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan
c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko
d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko
4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Szilard Nemeth
9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Jim Brennan
3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Szilard Nemeth
bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00