big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,067 Commits 2 Branches 2 Tags 264 MiB
5cc8c574d1
Commit Graph

6491 Commits

Author SHA1 Message Date
Szilard Nemeth
db8ae4b654 YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori 2022-03-09 19:36:09 +01:00
Szilard Nemeth
2ece95064b YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori 2022-03-09 19:29:53 +01:00
Viraj Jasani
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing (#4027) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-08 21:32:51 +09:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
9uapaw
851a4dc2d9 YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth 2022-03-07 09:01:18 +01:00
Szilard Nemeth
379baa5eb6 YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-03-03 16:44:12 +01:00
Szilard Nemeth
1c27c69f44 Revert "YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori" 
This reverts commit de526e166c.

With this commit, the patch of
https://issues.apache.org/jira/browse/YARN-10850 has been committed.
 2022-03-03 16:38:56 +01:00
Tamas Domok
22fe79cee3
YARN-11076. Upgrade jQuery version in Yarn UI2. (#4046) 
Change-Id: I3cb1677741df5a1978e83029443d4a2d5d7e3d7f
 2022-03-03 23:54:03 +09:00
Szilard Nemeth
356d337d1e YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok 2022-03-02 16:19:17 +01:00
9uapaw
9e475aede6 YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke 2022-03-02 11:16:24 +01:00
Szilard Nemeth
365375412f YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori 2022-02-22 12:26:35 +01:00
Szilard Nemeth
0463498adc YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori 2022-02-22 11:55:34 +01:00
Szilard Nemeth
de526e166c YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-02-17 21:05:01 +01:00
Szilard Nemeth
7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth
ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth
4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth
51001fc7bf YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok 2022-02-17 19:38:18 +01:00
Szilard Nemeth
48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth
bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth
d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
Akira Ajisaka
7fd90cdcbe
YARN-10788. TestCsiClient fails (#3989) 
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 01:13:13 +09:00
minni31
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
minni31
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741) 
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-02-01 12:18:54 +05:30
Akira Ajisaka
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2022-01-28 15:51:21 +09:00
Akira Ajisaka
02f6bad1ff Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936)" 
This reverts commit 1c01944f35.
 2022-01-28 00:36:25 +09:00
Wei-Chiu Chuang
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 00:04:01 +09:00
minni31
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse (#3766) 2022-01-25 09:25:18 -08:00
Andrew Chung
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container (#3779) 2022-01-24 08:03:36 -08:00
dependabot[bot]
dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui (#3890) 
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-20 21:44:10 +09:00
Gera Shegalov
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-17 16:20:48 +09:00
luoyuan3471
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Andras Gyori
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
minni31
3bf014d871
YARN-8859. Add audit logs for router service (#3607) 
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
 2022-01-03 10:38:59 +05:30
Cheng Pan
4db27acf06
YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 11:08:04 +09:00
Ashutosh Gupta
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 17:14:51 +09:00
Eric Payne
e2d6fd075d YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:05:39 +00:00
Viraj Jasani
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled (#3802) 2021-12-20 18:02:57 +08:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Szilard Nemeth
07141426e0
YARN-11050 (#3805) 2021-12-18 11:52:03 +08:00
Szilard Nemeth
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class (#3800). Contributed by Szilard Nemeth 2021-12-17 00:18:14 +01:00
Szilard Nemeth
aec9cdb467 YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok 2021-12-16 23:39:18 +01:00
Szilard Nemeth
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API (#3799). Contributed by Szilard Nemeth 2021-12-16 15:53:08 +01:00
Szilard Nemeth
6ea1685abd YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke 2021-12-16 00:01:09 +01:00
Viraj Jasani
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (#3796) 2021-12-15 16:47:51 +08:00
Szilard Nemeth
843f66f4dc Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke 2021-12-14 22:00:46 +01:00
Szilard Nemeth
364d38f00d YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke 2021-12-14 14:57:08 +01:00
Szilard Nemeth
a5bcf4c792 YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke 2021-12-13 21:57:46 +01:00
Szilard Nemeth
898055e204 YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke 2021-12-13 16:12:55 +01:00
Viraj Jasani
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown (#3768) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-13 14:03:32 +09:00
Szilard Nemeth
f544ef4a51 YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok 2021-12-10 15:09:53 +01:00
Szilard Nemeth
a0d8cde133 YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs 2021-12-09 17:51:44 +01:00
Andrew Chung
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy (#3717) 2021-12-08 10:02:10 -08:00
Szilard Nemeth
66a08e933a YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok 2021-12-08 16:41:35 +01:00
Szilard Nemeth
19430118c6 YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2021-12-08 16:03:42 +01:00
Viraj Jasani
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3749) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 16:52:22 +09:00
Szilard Nemeth
126079612c YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori 2021-12-07 15:55:16 +01:00
Szilard Nemeth
7284d23476 YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke 2021-12-07 15:43:55 +01:00
Szilard Nemeth
60440a836a YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok 2021-12-07 15:26:07 +01:00
Szilard Nemeth
7a8c2e0097 YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori 2021-12-07 08:51:03 +01:00
Szilard Nemeth
d75f2db2f9 YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2021-12-07 08:42:57 +01:00
Ashutosh Gupta
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-06 19:45:44 +09:00
Andras Gyori
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars (#3710) 
Contributed by Andras Gyori
 2021-12-03 16:44:58 +00:00
Viraj Jasani
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile (#3722) 2021-12-02 10:40:30 +08:00
Sushmasree-28
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files (#3735) 
Contributed by  sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
 2021-11-30 00:00:25 +05:30
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
GuoPhilipse
c9d64bad37
YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 17:48:54 +09:00
Ashutosh Gupta
08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 18:41:22 +09:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 (#3642) 2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources (#3646) 2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules (#3687) 2021-11-23 13:36:22 +09:00
Szilard Nemeth
b933f5f54b YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang 2021-11-15 16:02:18 +01:00
Szilard Nemeth
7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
minni31
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers (#3603) 2021-11-11 09:49:29 -08:00
Szilard Nemeth
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration (#3570). Contributed by Szilard Nemeth 2021-10-28 15:19:32 +02:00
Szilard Nemeth
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue (#3551) contributed by Szilard Nemeth 2021-10-27 19:03:45 +02:00
Szilard Nemeth
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler (#3581) Contributed by Szilard Nemeth 2021-10-27 17:13:49 +02:00
Jack
9cfd8d0a83 YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth 2021-10-23 14:47:09 +02:00
9uapaw
32ecaed9c3 YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori 2021-10-22 17:32:33 +02:00
Adam Antal
23772d946b YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal 2021-10-22 16:33:03 +02:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
Ahmed Hussein
d286994009 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-20 22:18:36 +00:00
Szilard Nemeth
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth 2021-10-20 15:52:37 +02:00
Andras Gyori
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori 2021-10-20 12:56:41 +02:00
Szilard Nemeth
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule (#3560) 
* YARN-10958. Initial commit

* Fix javadoc + behaviour

* Fix review comments

* fix checkstyle + blanks

* fix checkstyle + blanks

* Fix checkstyle + blanks
 2021-10-20 10:48:42 +02:00
9uapaw
616cea2e80 YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori 2021-10-19 13:06:45 +02:00
Szilard Nemeth
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth 2021-10-19 12:24:58 +02:00
lujiefsi
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk (#3552) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 15:24:15 +09:00
Viraj Jasani
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules (#3541) 2021-10-14 18:03:01 +09:00
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
Szilard Nemeth
8c271caa81 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-10-12 17:35:06 +02:00
Benjamin Teke
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-10-07 20:11:42 +02:00
9uapaw
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori 2021-10-07 17:09:38 +02:00
Tamas Domok
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-07 16:21:05 +02:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
9uapaw
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori 2021-10-01 04:20:36 +02:00
Tamas Domok
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-01 04:05:52 +02:00
Neil
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint (#3480) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-29 15:46:55 +09:00
Akshat Bordia
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics (#3439) 2021-09-28 09:51:18 -07:00
Tamas Domok
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-21 16:13:04 +02:00
9uapaw
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Adam Antal
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
TiborKovacsCloudera
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController (#3450) 2021-09-19 14:35:34 +02:00
Szilard Nemeth
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
adamantal
f5c76c8e31 YARN-10936. Log typo corrected. (#3449) 
Contributed by: Tibor Kovacs
 2021-09-17 08:27:51 +02:00
Eric Badger
43f0a34dd4 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-15 20:03:45 +00:00
Benjamin Teke
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources (#3418) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-14 18:05:40 +02:00
Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy (#3403) 
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic (#3390) 
- capacityConfigType update is extracted to a separate method
 - validation logic is extracted to a helper function
 - min resource must not be greater than max resource is now checked
   after the max resource is updated

Change-Id: I731c2639281721afed32c30854bafcf048d6ee28

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:30:44 +02:00
Weihao Zheng
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor (#3389) 
Contributed by Weihao Zheng
 2021-09-13 11:06:53 +08:00
Jack
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF (#3352) 
Contributed by Jie Wang <jie.wang@hulu.com>
 2021-09-13 10:54:11 +08:00
Benjamin Teke
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate (#3396) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 17:32:42 +02:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue (#3392) 2021-09-10 16:59:46 +02:00
Benjamin Teke
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic (#3407) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 16:48:58 +02:00
Tamas Domok
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue (#3402) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-10 15:57:46 +02:00
Akshat Bordia
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor (#3325) 2021-09-09 09:20:57 -07:00
Akshat Bordia
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report (#3252) 2021-09-08 10:36:56 -07:00
Szilard Nemeth
2ff3fc50e4 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-08 18:01:39 +02:00
Tamas Domok
5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3355) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-08 17:04:32 +02:00
Jack
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field (#3382) 
Co-authored-by: Jie Wang <jie.wang@hulu.com>
 2021-09-08 16:19:29 +02:00
Benjamin Teke
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario (#3339) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-08 16:11:04 +02:00
zhuqi
3024a47026
Add documentation for YARN-10623 auto refresh queue conf in CS (#3279) 2021-09-08 16:03:15 +02:00
Benjamin Teke
d9cb69853b
YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler 2021-09-08 15:43:57 +02:00
Szilard Nemeth
9c8fe1e512 YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke 2021-09-08 15:17:27 +02:00
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files (#3318) 
* YARN 10884 : Parse log files which has empty owner

* Removing the whitespace

* Added a test case for null user

* Fixed indentations

* Fixed the indentation for test cases

Co-authored-by: Swathi C <cswathi@microsoft.com>
 2021-09-07 21:37:51 +05:30
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-27 23:09:54 +02:00
Szilard Nemeth
e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287) 
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
 2021-08-17 14:18:55 +05:30
Gergely Pollak
44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
Szilard Nemeth
f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
9uapaw
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-27 19:35:20 -07:00
Szilard Nemeth
fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
dependabot[bot]
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-26 02:16:21 +09:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203) 
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135) 
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
 2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
9uapaw
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution (#3214) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-21 10:06:46 +08:00
Masatake Iwasaki
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 2021-07-20 11:52:52 +08:00