big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,792 Commits 2 Branches 2 Tags 264 MiB
6cc56d40f5
Commit Graph

6726 Commits

Author SHA1 Message Date
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-29 17:37:30 +09:00
GuoPhilipse
c9d64bad37
YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 17:48:54 +09:00
Ashutosh Gupta
08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 18:41:22 +09:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 (#3642) 2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources (#3646) 2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules (#3687) 2021-11-23 13:36:22 +09:00
Szilard Nemeth
b933f5f54b YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang 2021-11-15 16:02:18 +01:00
Szilard Nemeth
7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
minni31
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers (#3603) 2021-11-11 09:49:29 -08:00
Szilard Nemeth
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration (#3570). Contributed by Szilard Nemeth 2021-10-28 15:19:32 +02:00
Szilard Nemeth
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue (#3551) contributed by Szilard Nemeth 2021-10-27 19:03:45 +02:00
Szilard Nemeth
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler (#3581) Contributed by Szilard Nemeth 2021-10-27 17:13:49 +02:00
Jack
9cfd8d0a83 YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth 2021-10-23 14:47:09 +02:00
9uapaw
32ecaed9c3 YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori 2021-10-22 17:32:33 +02:00
Adam Antal
23772d946b YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal 2021-10-22 16:33:03 +02:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
Ahmed Hussein
d286994009 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-20 22:18:36 +00:00
Szilard Nemeth
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth 2021-10-20 15:52:37 +02:00
Andras Gyori
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori 2021-10-20 12:56:41 +02:00
Szilard Nemeth
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule (#3560) 
* YARN-10958. Initial commit

* Fix javadoc + behaviour

* Fix review comments

* fix checkstyle + blanks

* fix checkstyle + blanks

* Fix checkstyle + blanks
 2021-10-20 10:48:42 +02:00
9uapaw
616cea2e80 YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori 2021-10-19 13:06:45 +02:00
Szilard Nemeth
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth 2021-10-19 12:24:58 +02:00
lujiefsi
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk (#3552) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 15:24:15 +09:00
Viraj Jasani
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules (#3541) 2021-10-14 18:03:01 +09:00
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
Szilard Nemeth
8c271caa81 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-10-12 17:35:06 +02:00
Benjamin Teke
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-10-07 20:11:42 +02:00
9uapaw
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori 2021-10-07 17:09:38 +02:00
Tamas Domok
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-07 16:21:05 +02:00
Viraj Jasani
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
9uapaw
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori 2021-10-01 04:20:36 +02:00
Tamas Domok
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-10-01 04:05:52 +02:00
Neil
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint (#3480) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-29 15:46:55 +09:00
Akshat Bordia
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics (#3439) 2021-09-28 09:51:18 -07:00
Tamas Domok
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-21 16:13:04 +02:00
9uapaw
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Adam Antal
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
TiborKovacsCloudera
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController (#3450) 2021-09-19 14:35:34 +02:00
Szilard Nemeth
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
adamantal
f5c76c8e31 YARN-10936. Log typo corrected. (#3449) 
Contributed by: Tibor Kovacs
 2021-09-17 08:27:51 +02:00
Eric Badger
43f0a34dd4 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-15 20:03:45 +00:00
Benjamin Teke
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources (#3418) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-14 18:05:40 +02:00
Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy (#3403) 
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic (#3390) 
- capacityConfigType update is extracted to a separate method
 - validation logic is extracted to a helper function
 - min resource must not be greater than max resource is now checked
   after the max resource is updated

Change-Id: I731c2639281721afed32c30854bafcf048d6ee28

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:30:44 +02:00
Weihao Zheng
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor (#3389) 
Contributed by Weihao Zheng
 2021-09-13 11:06:53 +08:00
Jack
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF (#3352) 
Contributed by Jie Wang <jie.wang@hulu.com>
 2021-09-13 10:54:11 +08:00
Benjamin Teke
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate (#3396) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 17:32:42 +02:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue (#3392) 2021-09-10 16:59:46 +02:00
Benjamin Teke
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic (#3407) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-10 16:48:58 +02:00
Tamas Domok
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue (#3402) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-10 15:57:46 +02:00
Akshat Bordia
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor (#3325) 2021-09-09 09:20:57 -07:00
Akshat Bordia
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report (#3252) 2021-09-08 10:36:56 -07:00
Szilard Nemeth
2ff3fc50e4 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-08 18:01:39 +02:00
Tamas Domok
5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3355) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-08 17:04:32 +02:00
Jack
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field (#3382) 
Co-authored-by: Jie Wang <jie.wang@hulu.com>
 2021-09-08 16:19:29 +02:00
Benjamin Teke
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario (#3339) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-09-08 16:11:04 +02:00
zhuqi
3024a47026
Add documentation for YARN-10623 auto refresh queue conf in CS (#3279) 2021-09-08 16:03:15 +02:00
Benjamin Teke
d9cb69853b
YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler 2021-09-08 15:43:57 +02:00
Szilard Nemeth
9c8fe1e512 YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke 2021-09-08 15:17:27 +02:00
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files (#3318) 
* YARN 10884 : Parse log files which has empty owner

* Removing the whitespace

* Added a test case for null user

* Fixed indentations

* Fixed the indentation for test cases

Co-authored-by: Swathi C <cswathi@microsoft.com>
 2021-09-07 21:37:51 +05:30
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-27 23:09:54 +02:00
Szilard Nemeth
e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287) 
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
 2021-08-17 14:18:55 +05:30
Gergely Pollak
44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
Szilard Nemeth
f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
9uapaw
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-27 19:35:20 -07:00
Szilard Nemeth
fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
dependabot[bot]
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-26 02:16:21 +09:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203) 
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135) 
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
 2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
9uapaw
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution (#3214) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-21 10:06:46 +08:00
Masatake Iwasaki
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 2021-07-20 11:52:52 +08:00
zhuqi-lucas
0ac443b1f8 YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan. 2021-07-19 10:11:20 +08:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 (#3204) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-16 12:17:17 +09:00
Jim Brennan
632f64cadb YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 14:23:31 +00:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
Jim Brennan
dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Prabhu Joseph
6b4987f09c YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 2021-06-27 16:28:32 +05:30
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Peter Bacsko
0934e783cf YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori. 2021-06-24 20:15:10 +02:00
Masatake Iwasaki
11d144d228
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 2021-06-23 19:33:03 +09:00
Szilard Nemeth
6562391737 YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-06-16 18:55:09 +02:00
Szilard Nemeth
428478bbe2 YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori 2021-06-16 18:26:58 +02:00
Jim Brennan
9a6a11c452 YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 2021-06-15 18:58:42 +00:00
Szilard Nemeth
e31d06032b YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke 2021-06-14 22:33:04 +02:00
Szilard Nemeth
f0bdc422aa YARN-10816. Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun Parimi 2021-06-12 15:18:41 +02:00
Szilard Nemeth
7003997e36 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-06-12 14:49:52 +02:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Akira Ajisaka
9a17f97133
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. (#3068) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-10 14:29:47 +09:00
Viraj Jasani
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project (#3075) 2021-06-09 15:15:47 +09:00
Viraj Jasani
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-08 19:37:23 -07:00
Prabhu Josephraj
9445abb500 YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar 2021-06-08 20:37:40 +05:30
zhuqi-lucas
ec16b1d3b9 YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke. 2021-06-08 21:03:43 +08:00
Hideyuki Furue
9c7b8cf54e
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-06 07:05:29 +09:00
Szilard Nemeth
200eec8f2e YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko 2021-06-03 13:27:36 +02:00
Szilard Nemeth
2707f69251 YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak 2021-06-01 16:01:39 +02:00
Gergely Pollak
e9339aa376 YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth 2021-06-01 15:57:22 +02:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Jim Brennan
c4581827a9 YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 2021-05-25 18:07:28 +00:00
Szilard Nemeth
b86a6eb871 YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori 2021-05-25 18:27:53 +02:00
Szilard Nemeth
2541efa496 YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori 2021-05-25 13:55:59 +02:00
Akira Ajisaka
1cbcde04f3
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 2021-05-25 18:28:09 +09:00
Viraj Jasani
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project (#3033) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 18:10:20 +09:00
zhuqi-lucas
59172ada90 YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju. 2021-05-24 23:12:07 +08:00
Szilard Nemeth
6bb0892f6e YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke 2021-05-22 15:00:08 +02:00
Szilard Nemeth
1e44bdb84c YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke 2021-05-22 14:55:01 +02:00
Szilard Nemeth
d146ab3af7 YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed by Andras Gyori 2021-05-21 18:07:32 +02:00
Szilard Nemeth
43e77caa71 YARN-10779. Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl. Contributed by Peter Bacsko 2021-05-21 18:04:24 +02:00
Akira Ajisaka
c8073811b2
YARN-9279. Remove the old hamlet package. (#2986) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-21 12:40:50 +09:00
dependabot[bot]
f8b0063201
Bump node-sass (#3004) 
Bumps [node-sass](https://github.com/sass/node-sass) from 4.13.0 to 4.14.1.
- [Release notes](https://github.com/sass/node-sass/releases)
- [Changelog](https://github.com/sass/node-sass/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sass/node-sass/compare/v4.13.0...v4.14.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-18 20:14:18 -07:00
Peter Bacsko
eb72628e15 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-17 21:07:29 +02:00
Peter Bacsko
8891e5c028 YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju. 2021-05-17 13:30:12 +02:00
lujiefsi
d92a25b790
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-17 13:44:38 +09:00
kyoty
344cacce23
YARN-10545. Improve the readability of diagnostics log in yarn-ui2 web page. #2540 
Contributed by kyoty.
 2021-05-14 23:57:54 +08:00
zhuqi
e7f0e8073b YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu. 2021-05-14 13:36:07 +08:00
zhuqi
d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. (#2911) 
Contributed by Qi Zhu.
 2021-05-14 13:12:28 +08:00
dependabot[bot]
881ab4ed39
Bump underscore (#2997) 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.9.1 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.1...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-13 05:54:07 -07:00
dependabot[bot]
2f67a2646e
Bump acorn (#3003) 
Bumps [acorn](https://github.com/acornjs/acorn) from 5.7.3 to 5.7.4.
- [Release notes](https://github.com/acornjs/acorn/releases)
- [Commits](https://github.com/acornjs/acorn/compare/5.7.3...5.7.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-13 05:51:43 -07:00
Peter Bacsko
626be24c3e YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori. 2021-05-12 14:54:47 +02:00
Borislav Iordanov
91430889a5
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 13:31:48 -07:00
Akira Ajisaka
7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-06 23:37:45 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Eric Badger
803ac4b1a0 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 2021-04-29 17:42:13 +00:00
Siyao Meng
5b09ba75ec
YARN-10747. Bump YARN CSI protobuf version to 3.7.1 (#2946) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ib2a4bf775a5804641a91c6f9a486772c9d87c17e
 2021-04-28 12:44:13 +09:00
Peter Bacsko
a967ab06f2 YARN-10739. GenericEventHandler.printEventQueueDetails causes RM recovery to take too much time. Contributed by Qi Zhu. 2021-04-27 14:01:53 +02:00
Peter Bacsko
9166bfeb74 YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu. 2021-04-24 10:10:10 +02:00
Viraj Jasani
538ce9c354
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-04-25 23:48:39 -07:00
Eric Badger
6857a05d6a YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
 2021-04-23 22:02:04 +00:00
Jim Brennan
1cbe35946b YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 2021-04-23 21:35:09 +00:00
Szilard Nemeth
ebf6b14c67 YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 
This is required to highlight that an implementation at framework level is needed for the stop/start functionality to work.
 2021-04-23 22:49:26 +02:00
Peter Bacsko
14a84c47b0 YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja. 2021-04-23 17:38:47 +02:00
Szilard Nemeth
3b6b21e459 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 2021-04-23 16:17:29 +02:00
Szilard Nemeth
919daec36b YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak 2021-04-23 16:12:12 +02:00
Szilard Nemeth
f76a2a7606 YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko 2021-04-23 16:07:58 +02:00
Eric Badger
6cb90005a7 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 2021-04-20 17:34:49 +00:00
Viraj Jasani
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-17 20:47:44 +09:00
Miklos Gergely
ba3bc53f4e
YARN-10736. Fix GetApplicationsRequest JavaDoc. Contributed by Miklos Gergely. (#2906) 2021-04-14 17:16:30 +02:00
dependabot[bot]
6a9a5ae5fd
Bump y18n (#2843) 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-11 01:30:49 +09:00
Eric Badger
213d3deb26 YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 00:34:15 +00:00
Peter Bacsko
ca9aa91d10 YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori. 2021-04-08 12:42:48 +02:00
Szilard Nemeth
9cd69c20c4 YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori 2021-04-07 11:52:21 +02:00
Eric Badger
26b8f678b2 YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 01:16:14 +00:00
Peter Bacsko
158758c5bf YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu. 2021-04-01 16:09:52 +02:00
Peter Bacsko
9f1655baf2 YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu. 2021-04-01 11:39:40 +02:00
Peter Bacsko
a0deda1a77 YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 2021-04-01 09:21:15 +02:00
Borislav Iordanov
2c482fbacf HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:46:35 -07:00
stack
22961a615d Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" 
This reverts commit a2975d2153.
 2021-03-31 10:43:09 -07:00
stack
a2975d2153 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
 2021-03-31 10:40:20 -07:00
stack
5183aaeda2 Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693)" 
Revert to fix the summary message.

This reverts commit 9509bebf7f.
 2021-03-31 10:39:55 -07:00
Borislav Iordanov
9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes (#2693) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:07:42 -07:00
Szilard Nemeth
6fd0c661b6 YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak 2021-03-31 16:14:21 +02:00
Peter Bacsko
ff6ec20d84 YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu. 2021-03-31 10:55:14 +02:00
Akira Ajisaka
8ad77a26c9
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 2021-03-30 17:46:51 +09:00
Brahma Reddy Battula
413a4c3c05 YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy 2021-03-30 13:52:07 +05:30
Brahma Reddy Battula
6577bf1891 YARN-10439. addendum fix for shaded guva. 2021-03-30 13:48:40 +05:30
Brahma Reddy Battula
b61f52ec56 YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy 2021-03-30 09:51:47 +05:30
Brahma Reddy Battula
d0dcfc405c YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy 2021-03-30 09:46:12 +05:30
Brahma Reddy Battula
2d62dced4b YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy 2021-03-30 09:39:00 +05:30
Eric Badger
19e418c10d YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 2021-03-25 22:33:58 +00:00
Peter Bacsko
ceb75e1e2a YARN-10674. fs2cs should generate auto-created queue deletion properties. Contributed by Qi Zhu. 2021-03-24 08:15:06 +01:00
Jim Brennan
174f3a96b1 YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:21:45 +00:00
Jack Jiang
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Cyrus Jackson
cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications (#2674). Contributed by Cyrus Jackson 2021-03-19 15:49:05 +05:30
Eric Badger
3d4ea4b336 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-03-18 19:36:50 +00:00
Eric Badger
181b9ddc51 YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 2021-03-18 17:45:09 +00:00
Peter Bacsko
ce6bfd5718 YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu. 2021-03-18 13:40:16 +01:00
Peter Bacsko
38495af325 YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 2021-03-18 12:46:11 +01:00
Szilard Nemeth
a5745711dd YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak 2021-03-18 12:43:01 +01:00
Peter Bacsko
d7eeca4d0c YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu. 2021-03-18 11:49:16 +01:00
Eric Badger
49f89f1d3d YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 2021-03-17 18:11:37 +00:00
Peter Bacsko
3e58d5611d YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu. 2021-03-17 13:38:20 +01:00
Wilfred Spiegelenburg
f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it 
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.

Contributed by: Siddharth Ahuja
 2021-03-17 10:55:05 +11:00
Peter Bacsko
73394fabc7 YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu. 2021-03-16 16:09:12 +01:00
Peter Bacsko
b80588b688 YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu. 2021-03-16 15:23:27 +01:00
zhuqi
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. (#2760) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-16 13:50:29 +09:00
Eric Payne
aa4c17b9d7 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 2021-03-15 19:09:40 +00:00
Jim Brennan
5f067cf0f3 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 2021-03-12 17:17:31 +00:00
Szilard Nemeth
5db4c0bf70 YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak 2021-03-12 14:10:16 +01:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Brahma Reddy Battula
b2a565629d YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 2021-03-09 20:26:07 +05:30
Peter Bacsko
d5e035dbe1 YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-03-09 14:33:14 +01:00
Peter Bacsko
3851994cd6 Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu." 
This reverts commit 369f75b7a7.
 2021-03-09 14:32:02 +01:00
Peter Bacsko
369f75b7a7 YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu. 2021-03-09 14:28:23 +01:00
Peter Bacsko
20416bc62d YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth. 2021-03-09 11:34:37 +01:00
Peter Bacsko
e472ee2aa5 YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth. 2021-03-09 11:28:24 +01:00
Peter Bacsko
04cd3115ba YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu. 2021-03-08 12:29:44 +01:00
Peter Bacsko
c6b30a59dd YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth. 2021-03-05 23:13:35 +01:00
Eric Badger
9d088639bf YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.
 2021-03-05 19:46:40 +00:00
Peter Bacsko
e82e7c597a YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu. 2021-03-05 15:56:56 +01:00
Peter Bacsko
e7cad3811f YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-05 13:50:45 +01:00
Peter Bacsko
e19c00925f YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu. 2021-03-05 13:18:06 +01:00
Peter Bacsko
6699198b54 YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. 2021-03-04 17:18:35 +01:00
Peter Bacsko
a85aeee876 YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. 2021-03-04 12:23:11 +01:00
Neil
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 2021-03-04 14:52:58 +05:30
Peter Bacsko
7e8040e6ad YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. 2021-03-03 17:44:30 +01:00
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
Akira Ajisaka
3101c39b49
YARN-10656. Parsing error in CapacityScheduler.md (#2725) 
Reviewed-by: Peter Bacsko <pbacsko@apache.org>
 2021-02-27 12:46:36 +09:00
Peter Bacsko
947b50489d YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke. 2021-02-26 21:57:54 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Eric Badger
fcdb1e7fec YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu
 2021-02-26 19:03:59 +00:00
Jonathan Hung
ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan
c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko
a89c624002 YARN-10609. Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue). Contributed by Qi Zhu. 2021-02-24 20:41:35 +01:00
Eric Badger
47420ae3ed YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.
 2021-02-22 18:48:14 +00:00
Peter Bacsko
b722c52f4e YARN-10513. CS Flexible Auto Queue Creation RM UIv2 modifications. Contributed by Andras Gyori. 2021-02-22 12:55:45 +01:00
Eric Badger
4891e68c2b YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.
 2021-02-19 22:29:10 +00:00
Peter Bacsko
d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko
4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Szilard Nemeth
9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Jim Brennan
3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Szilard Nemeth
bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
98ca6afd17 YARN-10618. RM UI2 Application page shows the AM preempted containers instead of the nonAM ones. Contributed by Benjamin Teke 2021-02-11 09:14:37 +01:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Akira Ajisaka
52217fce3d
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-02-05 16:14:10 +09:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Szilard Nemeth
1ccba3734d YARN-10490. yarn top command not quitting completely with ctrl+c. Contributed by Agshin Kazimli 2021-01-21 20:30:19 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt
61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
He Xiaoqiao
ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein
890f2da624
YARN-10553. Refactor TestDistributedShell (#2581) 2021-01-09 08:39:21 +09:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Akira Ajisaka
ae4945fb2c
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 2021-01-06 12:12:46 +09:00
Masatake Iwasaki
5abeecd33f
YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586) 2021-01-05 13:38:34 +09:00
Neil
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Ahmed Hussein
513f1995ad
YARN-10334. Close clients in TestDistributedShell (#2571) 2020-12-28 11:09:10 -08:00
Eric Badger
4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.
 2020-12-21 23:10:31 +00:00
zhuqi
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. (#2563) 
Qi Zhu via Zhankun Tang
 2020-12-21 10:04:16 +08:00
Ahmed Hussein
7a88f45366
YARN-10536. Client in distributedShell swallows interrupt exceptions (#2554) 2020-12-17 15:13:28 -08:00
Akira Ajisaka
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka 2020-12-16 17:26:15 +01:00
Szilard Nemeth
df35c7f519 YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak 2020-12-15 17:30:14 +01:00
adamantal
3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
adamantal
0a45bd034e YARN-10520. Deprecated the residual nested class for the LCEResourceHandler (#2527). Contributed by Wanqiang Ji 2020-12-09 12:55:24 +01:00
zhuqi
d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler (#2494) 
Contributed by Qi Zhu.
 2020-12-09 19:48:39 +08:00
Eric Badger
7d3c8ef606 YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.
 2020-12-08 00:53:24 +00:00
Andrea Scarpino
9170eb566b
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-12-04 00:14:04 +09:00
adamantal
9969745343 YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy 2020-12-03 09:55:06 +01:00
Eric Payne
fa773a8326 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-01 22:06:47 +00:00
kevinzhao1661
4d2ae5b398
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-30 11:17:10 +09:00
Akira Ajisaka
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:32:10 +09:00
Akira Ajisaka
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488)" because JIRA ID is missing. 
This reverts commit ce5b3d7975.
 2020-11-26 11:30:53 +09:00
Akira Ajisaka
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:27:30 +09:00
Ankit Kumar
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-25 13:00:09 +09:00
Ahmed Hussein
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions (#2461) 2020-11-24 11:09:30 -08:00
Akira Ajisaka
c4ba0ab7df
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 2020-11-24 15:22:07 +09:00
Szilard Nemeth
0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth
5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Ahmed Hussein
0b2510ee1f
YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 2020-11-16 14:55:40 -06:00
Peter Bacsko
567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
Gabriel Medeiros Coelho
61f8c5767e
YARN-10480. replace href tags with ng-href (#2426) 2020-11-10 10:54:28 -08:00
Akira Ajisaka
4331c88352
HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-11-10 15:19:40 +09:00
Eric E Payne
55339c2bdd YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 2020-11-05 21:50:46 +00:00
Eric E Payne
0461a07c01 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 16:55:06 +00:00
neko
deea5d8f2b
YARN-10469. YARN-UI2 The accuracy of the percentage values in the same chart on the YARN 'Cluster OverView' page are inconsistent (#2401) 2020-11-02 17:52:58 +08:00
Szilard Nemeth
4890b855ec YARN-10420. Update CS MappingRule documentation with the new format and features. Contributed by Peter Bacsko 2020-10-31 15:17:01 +01:00
Surendra Singh Lilhore
7169ec4509
YARN-10442. RM should make sure node label file highly available. (#2390) 
* YARN-10442. RM should make sure node label file highly available.

* YARN-10442. Corrected method name.
            Added license header for file TestNodeLabelFileReplication.java
            Added Test code.

* YARN-10442. Changed property name and log.

* YARN-10442. Changed default replication for FS store file.
 2020-10-30 23:05:16 +05:30
Wangda Tan
c47c9fd65d YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. (Peter Bacsko via wangda) 
Change-Id: I518dc925187ce55e9d35a37ba20878c0f4e37e5c
 2020-10-30 09:49:48 -07:00
Jim Brennan
8ee6bc2518 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 19:49:18 +00:00
Jonathan Hung
bab5bf9743 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:33:18 -07:00
Sunil G
7435604a91 YARN-10453. Add partition resource info to get-node-labels and label-mappings api responses. Contributed Akhil PB. 2020-10-22 12:52:58 +05:30
Szilard Nemeth
31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Eric Badger
8b8c672780 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-15 17:57:43 +00:00
Szilard Nemeth
42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Jim Brennan
deb35a32ba YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:46:47 +00:00
Jim Brennan
b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Adam Antal
a1f7e760df YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-05 10:09:14 +02:00
Fei Hui
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Adam Antal
bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
Szilard Nemeth
a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Aryan Gupta
bbbfa7d415
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 2020-09-27 03:18:01 +09:00
Shubham Gupta
e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
Aryan Gupta
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-09-24 21:15:55 +09:00
Adam Antal
7ec16ac7f6 YARN-10443. Document options of logs CLI. Contributed by Ankit Kumar. 
Fixes #2325
 2020-09-23 17:42:56 +02:00
tkuramoto33
18f9d77c46
YARN-6754. Fair scheduler docs should explain meaning of weight=0 for a queue. (#2300) 2020-09-23 14:50:28 +09:00
Hui Fei
474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
Szilard Nemeth
eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Jim Brennan
90894ea641 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-14 21:19:14 +00:00
Eric E Payne
9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00
Szilard Nemeth
773ac799c6 YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko 2020-09-09 15:12:58 +02:00
Brahma Reddy Battula
43572fc7f8 YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 2020-09-09 17:08:13 +05:30
Peter Bacsko
aba4a506d6 YARN-10424. Adapt existing AppName and UserGroupMapping unittests to ensure backwards compatibility. Contributed by Benjamin Teke. 2020-09-09 09:41:11 +02:00
Peter Bacsko
c4fb4044b2 YARN-10415. Create a group matcher which checks ALL groups of the user. Contributed by Gergely Pollak. 2020-09-08 10:57:00 +02:00
Adam Antal
34fe74da0e YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 2020-09-07 11:44:09 +02:00
Peter Bacsko
360bbcd3bc YARN-10411. Create an allowCreate flag for MappingRuleAction. Contributed by Gergely Pollak. 2020-09-07 11:39:03 +02:00
Adam Antal
dd6c66556e YARN-9136. getNMResourceInfo NodeManager REST API method is not documented 2020-09-07 10:29:20 +02:00
Peter Bacsko
1841a5bb03 YARN-10376. Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak. 2020-09-05 10:00:33 +02:00
Szilard Nemeth
696e4fe50e Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja 2020-09-04 09:59:54 +02:00
Peter Bacsko
5c15815773 YARN-10375. CS Mapping rule config parser should return MappingRule objects. Contributed by Gergely Pollak. 2020-09-03 18:54:22 +02:00
Peter Bacsko
288dab7563 YARN-10372. Create MappingRule class to represent each CS mapping rule. Contributed by Gergely Pollak 2020-09-03 15:02:48 +02:00
Masatake Iwasaki
5e12dc51f0
YARN-10419. Javadoc error in hadoop-yarn-server-common module. (#2271) 2020-09-03 16:17:52 +09:00
Peter Bacsko
c7dab2b23e YARN-10374. Create Actions for CS mapping rules. Contributed by Gergely Pollak. 2020-09-01 17:38:49 +02:00
Adam Antal
73a0d149e2 YARN-10386. Addendum patch: Fixing ASF Warnings. Contributed by Peter Bacsko. 2020-09-01 14:39:24 +02:00
Szilard Nemeth
4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Szilard Nemeth
56a5c360a1 YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak 2020-08-29 21:34:55 +02:00
Szilard Nemeth
f4f872b778 YARN-10371. Create variable context class for CS queue mapping rules. Contributed by Gergely Pollak 2020-08-29 17:31:48 +02:00
Szilard Nemeth
a888d580d8 YARN-10409. Improve MockQueueHierarchyBuilder to detect queue ambiguity. Contributed by Gergely Pollak 2020-08-29 15:24:50 +02:00
Szilard Nemeth
44542863f4 YARN-10408. Extract MockQueueHierarchyBuilder to a separate class. Contributed by Gergely Pollak 2020-08-28 17:35:45 +02:00
Szilard Nemeth
6e3d7050a9 YARN-10386. Create new JSON schema for Placement Rules. Contributed by Peter Bacsko and Benjamin Teke 2020-08-28 17:09:31 +02:00
Prabhu Joseph
75db5526b5 YARN-1806. Add ThreadDump Option in YARN UI2 to fetch for running containers 
Contributed by Siddharth Ahuja. Reviewed by Akhil PB.
 2020-08-26 13:08:17 +05:30
Adam Antal
82a7505646 YARN-10304. Create an endpoint for remote application log directory path query. Contributed by Andras Gyori 2020-08-25 13:29:12 +02:00
Adam Antal
c049296235 YARN-10106. Yarn logs CLI filtering by application attempt. Contributed by Hudáky Márton Gyula 2020-08-25 09:53:39 +02:00
Prabhu Joseph
5c927eb550 YARN-10360. Support Multi Node Placement in SingleConstraintAppPlacementAllocator 
Reviewed by Sunil G.
 2020-08-24 15:00:02 +05:30
Zhengbo Li
9b9f7ea16a
YARN-10399 Refactor NodeQueueLoadMonitor class to make it extendable (#2228) 
Refactor NodeQueueLoadMonitor class to make it extendable
 2020-08-19 10:14:50 -07:00
Sunil G
82ec28f442 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-08-19 11:54:48 +05:30
Jim Brennan
4a7deae478 YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger 2020-08-17 16:41:15 +00:00
hemanthboyina
86bbd38c8d YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T. 2020-08-13 23:44:42 +05:30
Prabhu Joseph
6c2ce3d56b YARN-10389. Option to override RMWebServices with custom WebService class 
Contributed by Tanu Ajmera. Reviewed by Bilwa ST and Sunil G.
 2020-08-11 15:46:47 +05:30
Prabhu Joseph
5e0f879779 YARN-10364. Fix logic of queue capacity is absolute resource or percentage. 
Contributed by Bilwa ST. Reviewed by Sunil G.
 2020-08-08 22:33:58 +05:30
Jonathan Hung
64753addba YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:01 -07:00
bibinchundatt
81da221c75 YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 2020-08-07 08:36:52 +05:30
Prabhu Joseph
c7e71a6c0b YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.
 2020-08-06 13:19:52 +05:30
Eric Badger
5edd8b925e YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 18:47:49 +00:00
Prabhu Joseph
0277856738 YARN-10377. Fix filter index to show apps while clicking on queue in RM UI 
Contributed by Tarun Parimi.
 2020-08-04 23:10:45 +05:30
Prabhu Joseph
aa5afa72c5 YARN-10381. Add application attempt state in AppAttempts RM REST API 
Contributed by Siddharth Ahuja. Reviewed by Bilwa ST.
 2020-08-04 23:07:28 +05:30
kevinzhao1661
ab2b3df2de
YARN-10383. YarnCommands.md is inconsistent with the source code (#2177) 2020-08-04 12:02:16 +09:00
Brahma Reddy Battula
eac558380f YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 2020-08-03 12:54:36 +05:30
bibinchundatt
5323e83edf YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 2020-08-01 13:03:46 +05:30
Eric E Payne
e0c9653166 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 15:30:22 +00:00
bibinchundatt
5d8600e80a YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 2020-07-29 23:26:08 +05:30
Jonathan Hung
3eaf62726f YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:43:19 -07:00
bibinchundatt
5dadf963d3 YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. Contributed by Pranjal Protim Borah. 2020-07-28 11:55:47 +05:30
Prabhu Joseph
60a254621a YARN-10366. Fix Yarn rmadmin help message shows two labels for one node for --replaceLabelsOnNode. 
Contributed by Tanu Ajmera.
 2020-07-27 12:23:21 +05:30
Akira Ajisaka
d02be17a26
YARN-10362. Javadoc for TimelineReaderAuthenticationFilterInitializer is broken. Contributed by Xieming Li. 2020-07-27 01:55:04 +09:00
Eric Badger
ac5f21dbef YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.
 2020-07-24 22:35:16 +00:00
Prabhu Joseph
247eb0979b YARN-10319. Record Last N Scheduler Activities from ActivitiesManager 
Reviewed by Tao Yang and Adam Antal.
 2020-07-24 11:43:31 +05:30
bibinchundatt
bfcd775381 YARN-10315. Avoid sending RMNodeResourceupdate event if resource is same. Contributed by Sushil Ks. 2020-07-23 16:41:35 +05:30
Masatake Iwasaki
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Eric Badger
736bed6d6d [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan
 2020-07-20 18:08:27 +00:00
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Prabhu Joseph
cc71d50b21 YARN-10339. Fix TimelineClient in NodeManager failing when Simple Http Auth used in Secure Cluster 
Contributed by Tarun Parimi.
 2020-07-16 23:36:49 +05:30
Ahmed Hussein
1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 11:39:06 -05:00
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Bilwa S T
8980c23fc2
YARN-10350. TestUserGroupMappingPlacementRule fails 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-07-15 14:31:34 +09:00
Eric Badger
48f90115b5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan
 2020-07-13 23:10:39 +00:00
Eric E Payne
0427100b75 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-07-13 18:57:50 +00:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Sunil G
5dd270e208 YARN-10333. YarnClient obtain Delegation Token for Log Aggregation Path. Contributed by Prabhu Joseph. 2020-07-09 12:50:25 +05:30
Brahma Reddy Battula
dfe60392c9 YARN-10341. Yarn Service Container Completed event doesn't get processed. Contributed by Bilwa S T. 2020-07-09 12:34:52 +05:30
Akira Ajisaka
10d218934c
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) 2020-07-09 10:29:47 +05:30
Prabhu Joseph
3a4d05b850 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.
 2020-07-08 12:43:20 +05:30
Prabhu Joseph
2bbd00dff4 YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.
 2020-07-07 17:10:59 +05:30
Akira Ajisaka
639acb6d89
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein. 2020-07-06 16:08:36 +09:00
Szilard Nemeth
04abd0eb17 YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 14:10:55 +02:00
Szilard Nemeth
9b5557a9e8 YARN-10325. Document max-parallel-apps for Capacity Scheduler. Contributed by Peter Bacsko 2020-07-01 13:41:30 +02:00
Eric Yang
e8dc862d38 YARN-9809. Added node manager health status to resource manager registration call. 
Contributed by Eric Badger via eyang
 2020-06-30 11:39:16 -07:00
Akira Ajisaka
cd188ea9f0
YARN-10331. Upgrade node.js to 10.21.0. (#2106) 2020-06-30 16:52:57 +09:00
Eric E Payne
74fc13cf91 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 18:39:53 +00:00
Eric Yang
0be26811f3 YARN-10328. Fixed ZK Curator NodeExists exception in YARN service AM logs 
Contributed by Bilwa S T via eyang
 2020-06-29 09:21:24 -07:00
Brahma Reddy Battula
2c03524fa4 YARN-6526. Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call. COntributed by Bilwa S T. 2020-06-26 20:43:27 +05:30
Szilard Nemeth
e0c1d8a969 YARN-10327. Remove duplication of checking for invalid application ID in TestLogsCLI. Contributed by Marton Hudaky 2020-06-26 13:21:55 +02:00
Szilard Nemeth
6459cc73e6 YARN-10318. ApplicationHistory Web UI incorrect column indexing. Contributed by Andras Gyori 2020-06-26 13:15:14 +02:00
Szilard Nemeth
33d3df7ee2 YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy. Contributed by Szilard Nemeth 2020-06-26 11:14:24 +02:00
Szilard Nemeth
6a8fd73b27 YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed by Marton Hudaky 2020-06-25 17:27:58 +02:00
Szilard Nemeth
03f855e3e7 YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko 2020-06-23 12:13:04 +02:00
Prabhu Joseph
ce1008fe61 YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios 
Contributed by Szilard Nemeth. Reviewed by Gergely Pollak.
 2020-06-22 12:49:14 +05:30
Surendra Singh Lilhore
b2facc84a1 YARN-9460. QueueACLsManager and ReservationsACLManager should not use instanceof checks. Contributed by Bilwa S T. 2020-06-20 20:09:31 +05:30
Szilard Nemeth
469841446f YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-19 14:50:24 +02:00
Eric Yang
89689c52c3 YARN-10308. Update javadoc and variable names for YARN service. 
Contributed by Bilwa S T via eyang
 2020-06-17 09:04:26 -07:00
Szilard Nemeth
5b1a56f9f1 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:34:40 +02:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Tao Yang
7c4de59fc1 YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph. 2020-06-12 22:10:25 +08:00
Eric Badger
fed6fecd3a YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan.
 2020-06-11 21:02:41 +00:00
Eric E Payne
56247db302 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 2020-06-09 18:43:16 +00:00
Szilard Nemeth
8abff5151a YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-05 15:00:39 +02:00
Szilard Nemeth
8b146c17b3 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-05 11:38:32 +02:00
Szilard Nemeth
a4835db95a YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-05 10:11:40 +02:00
Szilard Nemeth
b5efdea4fd YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:29:15 +02:00
Szilard Nemeth
aa6d13455b YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-02 15:53:48 +02:00
Szilard Nemeth
e0a0741ac8 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-02 15:42:39 +02:00
Szilard Nemeth
593af878c0 YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-27 13:05:52 +02:00
Szilard Nemeth
fb030f3239 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit ac4540dd8e.
Reverting because of missing Jira-ID in the commit message.
 2020-05-27 13:04:48 +02:00
Jonathan Hung
c30c23cb66 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-05-26 16:29:42 -07:00
Szilard Nemeth
ac4540dd8e FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-21 13:11:44 +02:00
Eric Yang
726b8e324b YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts. 
Contributed by Bilwa S T via eyang
 2020-05-20 09:23:56 -07:00
Prabhu Joseph
cef0756929 YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient. 
Contributed by Bilwa S T. Reviewed by Akira Ajisaka.
 2020-05-20 11:36:52 +05:30
Akira Ajisaka
b65815d691
Revert "YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient." 
This reverts commit 7836bc4c35.
 2020-05-18 16:29:07 +09:00
Prabhu Joseph
7836bc4c35 YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient. 
Contributed by Bilwa S T. Reviewed by Brahma Reddy Battula.
 2020-05-14 19:40:42 +05:30
Prabhu Joseph
6ce295b787 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.
 2020-05-14 16:52:11 +05:30
Surendra Singh Lilhore
1958cb7c2b YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:09:34 +05:30
Ayush Saxena
0918433b4d YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:36:20 +05:30
Inigo Goiri
108ecf992f YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 2020-05-13 10:04:12 -07:00
Prabhu Joseph
450e5aa9dd YARN-10154. Addendum Patch which fixes below bugs 
1. RM fails to start when LeafQueueTemplate max capacity is not specified.
2. Job stuck in ACCEPTED state with DominantResourceCalculator as Queue
   Capacity is set to NaN during RM startup with clusterResource is zero.

Reviewed by Sunil G and Manikandan R.
 2020-05-13 14:35:37 +05:30
Jonathan Hung
fff1d2c122 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 2020-05-12 10:48:54 -07:00
Elixir Kook
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007) 
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
 2020-05-12 10:50:04 -05:00
Inigo Goiri
35010120fb YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 2020-05-06 13:22:54 -07:00
Ahmed Hussein
92e3ebb401 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-05-06 12:27:17 -05:00
Szilard Nemeth
cb6399c109 YARN-10257. FS-CS converter: skip increment properties for mem/vcores and fix DRF check. Contributed by Peter Bacsko 2020-05-05 16:00:43 +02:00
Szilard Nemeth
0debe55d6c YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-05 13:11:28 +02:00
Ahmed Hussein
f5081a9a5d YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-05-04 10:13:51 -05:00
Surendra Singh Lilhore
257b04ce4e YARN-6553. Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests. Contributed by Bilwa S T. 2020-05-02 12:43:20 +05:30
Inigo Goiri
d125d39108 YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by Bilwa S T. 2020-04-29 11:23:58 -07:00
Szilard Nemeth
410c605aec YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 2020-04-29 15:53:30 +02:00
Szilard Nemeth
ab36429559 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 2020-04-28 18:53:28 +02:00
Akira Ajisaka
6ef01646ba
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-04-28 05:35:36 +09:00