big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,372 Commits 2 Branches 2 Tags 264 MiB
a73ff6915a
Commit Graph

6097 Commits

Author SHA1 Message Date
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
Benjamin Teke
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 19:15:27 +02:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194) 
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
 2021-07-29 14:55:39 +05:30
Szilard Nemeth
f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth
1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
9uapaw
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-27 19:35:20 -07:00
Szilard Nemeth
fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
dependabot[bot]
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-26 02:16:21 +09:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203) 
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135) 
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
 2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
9uapaw
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution (#3214) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
 2021-07-21 10:06:46 +08:00
Masatake Iwasaki
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 2021-07-20 11:52:52 +08:00
zhuqi-lucas
0ac443b1f8 YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan. 2021-07-19 10:11:20 +08:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 (#3204) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-16 12:17:17 +09:00
Jim Brennan
632f64cadb YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 14:23:31 +00:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
Jim Brennan
dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Prabhu Joseph
6b4987f09c YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 2021-06-27 16:28:32 +05:30
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Peter Bacsko
0934e783cf YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori. 2021-06-24 20:15:10 +02:00
Masatake Iwasaki
11d144d228
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 2021-06-23 19:33:03 +09:00
Szilard Nemeth
6562391737 YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-06-16 18:55:09 +02:00
Szilard Nemeth
428478bbe2 YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori 2021-06-16 18:26:58 +02:00
Jim Brennan
9a6a11c452 YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 2021-06-15 18:58:42 +00:00
Szilard Nemeth
e31d06032b YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke 2021-06-14 22:33:04 +02:00
Szilard Nemeth
f0bdc422aa YARN-10816. Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun Parimi 2021-06-12 15:18:41 +02:00
Szilard Nemeth
7003997e36 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-06-12 14:49:52 +02:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Akira Ajisaka
9a17f97133
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. (#3068) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-10 14:29:47 +09:00
Viraj Jasani
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project (#3075) 2021-06-09 15:15:47 +09:00
Viraj Jasani
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-08 19:37:23 -07:00
Prabhu Josephraj
9445abb500 YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar 2021-06-08 20:37:40 +05:30
zhuqi-lucas
ec16b1d3b9 YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke. 2021-06-08 21:03:43 +08:00
Hideyuki Furue
9c7b8cf54e
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-06 07:05:29 +09:00
Szilard Nemeth
200eec8f2e YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko 2021-06-03 13:27:36 +02:00
Szilard Nemeth
2707f69251 YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak 2021-06-01 16:01:39 +02:00
Gergely Pollak
e9339aa376 YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth 2021-06-01 15:57:22 +02:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Jim Brennan
c4581827a9 YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 2021-05-25 18:07:28 +00:00
Szilard Nemeth
b86a6eb871 YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori 2021-05-25 18:27:53 +02:00
Szilard Nemeth
2541efa496 YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori 2021-05-25 13:55:59 +02:00
Akira Ajisaka
1cbcde04f3
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 2021-05-25 18:28:09 +09:00
Viraj Jasani
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project (#3033) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 18:10:20 +09:00
zhuqi-lucas
59172ada90 YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju. 2021-05-24 23:12:07 +08:00
Szilard Nemeth
6bb0892f6e YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke 2021-05-22 15:00:08 +02:00
Szilard Nemeth
1e44bdb84c YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke 2021-05-22 14:55:01 +02:00
Szilard Nemeth
d146ab3af7 YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed by Andras Gyori 2021-05-21 18:07:32 +02:00
Szilard Nemeth
43e77caa71 YARN-10779. Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl. Contributed by Peter Bacsko 2021-05-21 18:04:24 +02:00
Akira Ajisaka
c8073811b2
YARN-9279. Remove the old hamlet package. (#2986) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-21 12:40:50 +09:00
dependabot[bot]
f8b0063201
Bump node-sass (#3004) 
Bumps [node-sass](https://github.com/sass/node-sass) from 4.13.0 to 4.14.1.
- [Release notes](https://github.com/sass/node-sass/releases)
- [Changelog](https://github.com/sass/node-sass/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sass/node-sass/compare/v4.13.0...v4.14.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-18 20:14:18 -07:00
Peter Bacsko
eb72628e15 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-17 21:07:29 +02:00
Peter Bacsko
8891e5c028 YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju. 2021-05-17 13:30:12 +02:00
lujiefsi
d92a25b790
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-17 13:44:38 +09:00
kyoty
344cacce23
YARN-10545. Improve the readability of diagnostics log in yarn-ui2 web page. #2540 
Contributed by kyoty.
 2021-05-14 23:57:54 +08:00
zhuqi
e7f0e8073b YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu. 2021-05-14 13:36:07 +08:00
zhuqi
d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. (#2911) 
Contributed by Qi Zhu.
 2021-05-14 13:12:28 +08:00
dependabot[bot]
881ab4ed39
Bump underscore (#2997) 
Bumps [underscore](https://github.com/jashkenas/underscore) from 1.9.1 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases)
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.1...1.13.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-13 05:54:07 -07:00
dependabot[bot]
2f67a2646e
Bump acorn (#3003) 
Bumps [acorn](https://github.com/acornjs/acorn) from 5.7.3 to 5.7.4.
- [Release notes](https://github.com/acornjs/acorn/releases)
- [Commits](https://github.com/acornjs/acorn/compare/5.7.3...5.7.4)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
 2021-05-13 05:51:43 -07:00
Peter Bacsko
626be24c3e YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori. 2021-05-12 14:54:47 +02:00
Borislav Iordanov
91430889a5
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 13:31:48 -07:00
Akira Ajisaka
7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-06 23:37:45 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Eric Badger
803ac4b1a0 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 2021-04-29 17:42:13 +00:00
Siyao Meng
5b09ba75ec
YARN-10747. Bump YARN CSI protobuf version to 3.7.1 (#2946) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ib2a4bf775a5804641a91c6f9a486772c9d87c17e
 2021-04-28 12:44:13 +09:00
Peter Bacsko
a967ab06f2 YARN-10739. GenericEventHandler.printEventQueueDetails causes RM recovery to take too much time. Contributed by Qi Zhu. 2021-04-27 14:01:53 +02:00
Peter Bacsko
9166bfeb74 YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu. 2021-04-24 10:10:10 +02:00
Viraj Jasani
538ce9c354
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-04-25 23:48:39 -07:00
Eric Badger
6857a05d6a YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
 2021-04-23 22:02:04 +00:00
Jim Brennan
1cbe35946b YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 2021-04-23 21:35:09 +00:00
Szilard Nemeth
ebf6b14c67 YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 
This is required to highlight that an implementation at framework level is needed for the stop/start functionality to work.
 2021-04-23 22:49:26 +02:00
Peter Bacsko
14a84c47b0 YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja. 2021-04-23 17:38:47 +02:00
Szilard Nemeth
3b6b21e459 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 2021-04-23 16:17:29 +02:00
Szilard Nemeth
919daec36b YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak 2021-04-23 16:12:12 +02:00
Szilard Nemeth
f76a2a7606 YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko 2021-04-23 16:07:58 +02:00
Eric Badger
6cb90005a7 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 2021-04-20 17:34:49 +00:00
Viraj Jasani
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-17 20:47:44 +09:00
Miklos Gergely
ba3bc53f4e
YARN-10736. Fix GetApplicationsRequest JavaDoc. Contributed by Miklos Gergely. (#2906) 2021-04-14 17:16:30 +02:00
dependabot[bot]
6a9a5ae5fd
Bump y18n (#2843) 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-11 01:30:49 +09:00
Eric Badger
213d3deb26 YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 00:34:15 +00:00
Peter Bacsko
ca9aa91d10 YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori. 2021-04-08 12:42:48 +02:00
Szilard Nemeth
9cd69c20c4 YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori 2021-04-07 11:52:21 +02:00
Eric Badger
26b8f678b2 YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 01:16:14 +00:00
Peter Bacsko
158758c5bf YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu. 2021-04-01 16:09:52 +02:00
Peter Bacsko
9f1655baf2 YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu. 2021-04-01 11:39:40 +02:00
Peter Bacsko
a0deda1a77 YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 2021-04-01 09:21:15 +02:00
Borislav Iordanov
2c482fbacf HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:46:35 -07:00
stack
22961a615d Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" 
This reverts commit a2975d2153.
 2021-03-31 10:43:09 -07:00
stack
a2975d2153 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
 2021-03-31 10:40:20 -07:00
stack
5183aaeda2 Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693)" 
Revert to fix the summary message.

This reverts commit 9509bebf7f.
 2021-03-31 10:39:55 -07:00
Borislav Iordanov
9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes (#2693) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:07:42 -07:00
Szilard Nemeth
6fd0c661b6 YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak 2021-03-31 16:14:21 +02:00
Peter Bacsko
ff6ec20d84 YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu. 2021-03-31 10:55:14 +02:00