Commit Graph

6052 Commits

Author SHA1 Message Date
Jim Brennan
3b1cde2f42 YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu.
(cherry picked from commit 1cbe35946b)
2021-04-23 21:38:02 +00:00
Eric Badger
1960924d07 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu
(cherry picked from commit 6cb90005a7)
2021-04-20 17:46:05 +00:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko
(cherry picked from commit 31477a224f)
2021-04-16 20:21:01 +00:00
dependabot[bot]
d0b02a23f3 Bump y18n
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit 1a90fc4b50068865359809cd437e45595c397195)
2021-04-11 01:31:27 +09:00
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom
resourceType. Contributed by Qi Zhu.
2021-04-09 17:51:01 +00:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor.
Contributed by Jim Brennan.
2021-04-06 23:34:35 +00:00
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
2021-03-31 10:50:28 -07:00
Akira Ajisaka
c96fbb9199
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai.
(cherry picked from commit 8ad77a26c9)
2021-03-30 17:47:54 +09:00
Brahma Reddy Battula
5358313f97 YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy
(cherry picked from commit 413a4c3c05)
2021-03-30 13:52:35 +05:30
Brahma Reddy Battula
bac1326e4e YARN-10439. addendum fix for shaded guva.
(cherry picked from commit 6577bf1891)
2021-03-30 13:49:23 +05:30
Brahma Reddy Battula
616a41ee32 YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy
(cherry picked from commit b61f52ec56)
2021-03-30 09:52:28 +05:30
Brahma Reddy Battula
5181b2004b YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy
(cherry picked from commit d0dcfc405c)
2021-03-30 09:46:52 +05:30
Brahma Reddy Battula
e9d8f16a70 YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy
(cherry picked from commit 2d62dced4b)
2021-03-30 09:40:06 +05:30
Eric Badger
65bba8c3ed YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu.
(cherry picked from commit 19e418c10d)
2021-03-25 22:35:19 +00:00
Jim Brennan
78bddd0d9f YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T.
(cherry picked from commit 174f3a96b1)
2021-03-23 18:23:50 +00:00
Eric Badger
0e37da334b YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu.
(cherry picked from commit 181b9ddc51)
2021-03-18 17:47:18 +00:00
Eric Badger
8c82e3507d YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu.
(cherry picked from commit 38495af325)
2021-03-18 17:47:10 +00:00
Eric Badger
cd417f17ae YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu.
(cherry picked from commit 49f89f1d3d)
2021-03-17 18:16:59 +00:00
Eric Badger
1022ed91ac YARN-10495. make the rpath of container-executor configurable. Contributed by
angerszhu.

(cherry picked from commit 7d3c8ef606)
2021-03-15 20:01:24 +00:00
Eric Payne
f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T
(cherry picked from commit aa4c17b9d7)
2021-03-15 19:14:19 +00:00
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu.
(cherry picked from commit 5f067cf0f3)
2021-03-12 17:20:37 +00:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753)
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
2021-03-11 14:57:03 +09:00
Brahma Reddy Battula
f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy.
(cherry picked from commit b2a565629d)
2021-03-09 20:27:07 +05:30
Peter Bacsko
eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Peter Bacsko
066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
Eric Badger
1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim
Brennan.

(cherry picked from commit 9d088639bf)
2021-03-05 19:49:49 +00:00
Neil
0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie
(cherry picked from commit d615e2d3bd)
2021-03-04 14:54:28 +05:30
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein
(cherry picked from commit e04bcb3a06)
2021-02-26 19:56:07 +00:00
Eric Badger
e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by
Qi Zhu

(cherry picked from commit fcdb1e7fec)
2021-02-26 19:07:59 +00:00
Jonathan Hung
be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
2021-02-25 15:09:33 -08:00
Jim Brennan
db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne
(cherry picked from commit c373da9f88)
2021-02-25 16:48:46 +00:00
Eric Badger
305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by
Qi Zhu.

(cherry picked from commit 47420ae3ed)
2021-02-22 18:59:45 +00:00
Eric Badger
27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
2021-02-19 23:25:25 +00:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T.
(cherry picked from commit 35010120fb)
2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T.
(cherry picked from commit 108ecf992f)
2021-02-18 20:32:32 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T.
(cherry picked from commit 5323e83edf)
2021-02-18 20:23:56 +05:30
Prabhu Joseph
72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver.
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0b)
2021-02-18 14:25:16 +05:30
Prabhu Joseph
0c46ab51b5 YARN-8047. RMWebApp make external class pluggable.
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850)
2021-02-18 13:59:50 +05:30
Jim Brennan
457c8404b9 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger
(cherry picked from commit 3fcc7a55b6)
2021-02-16 16:47:46 +00:00
Masatake Iwasaki
4468378e4b YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619)
(cherry picked from commit f9a073c6c1)
2021-02-11 20:26:09 +00:00
Jim Brennan
d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger.
(cherry picked from commit c22c77af43)
2021-02-05 17:37:04 +00:00
Akira Ajisaka
bdd22b61c2
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546)
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 52217fce3d)
2021-02-05 16:15:06 +09:00
bibinchundatt
1520b84b36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal
(cherry picked from commit 8bc2dfbf36)
2021-01-22 08:30:12 +05:30
Eric Badger
7fc624aabc YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan.
(cherry picked from commit 768e2f42ba)
2021-01-13 23:43:15 +00:00
Eric Payne
4518ece4c9 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan)
(cherry picked from commit d8fadec944)
2021-01-13 17:17:07 +00:00
Ahmed Hussein
98690134d5 YARN-10334. Close clients in TestDistributedShell (#2571)
(cherry picked from commit 513f1995ad)
2021-01-09 21:18:00 +09:00
Ahmed Hussein
a40eabd717 YARN-10536. Client in distributedShell swallows interrupt exceptions (#2554)
(cherry picked from commit 7a88f45366)
2021-01-09 21:18:00 +09:00
Masatake Iwasaki
35740a25d9 YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586)
(cherry picked from commit 5abeecd33f)
2021-01-09 21:18:00 +09:00
He Xiaoqiao
e95ee67632
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:10 +08:00
Neil
cd5ee0014f YARN-10541. capture the performance metrics of ZKRMStateStore (#2568)
(cherry picked from commit fa4cf91b57)
2021-01-08 10:38:08 -08:00
Szilard Nemeth
f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
srinivasst
98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564)
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
2021-01-08 10:57:37 +05:30
Akira Ajisaka
3748c258f9
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592)
(cherry picked from commit ae4945fb2c)
2021-01-06 12:14:09 +09:00
Eric Badger
1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
2020-12-21 23:11:47 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538)
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
2020-12-11 13:58:02 +09:00
Akira Ajisaka
b84bc63aa2 HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434)
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-10 05:51:45 +05:30
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Andrea Scarpino
c5b9c5dfe5
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512)
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b)
2020-12-04 00:16:45 +09:00
Eric Payne
1fd6d81617 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth)
(cherry picked from commit fa773a8326)
2020-12-01 22:51:20 +00:00
kevinzhao1661
7d9c6edadf
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398)
2020-11-30 11:17:59 +09:00
Akira Ajisaka
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287)
2020-11-26 11:34:39 +09:00
Akira Ajisaka
bb4c6db9b3
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi.
(cherry picked from commit c4ba0ab7df)
2020-11-24 15:22:51 +09:00
Ahmed Hussein
bc634c077b YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein
(cherry picked from commit 0b2510ee1f)
2020-11-16 21:09:36 +00:00
Gabriel Medeiros Coelho
9a3f2bb545 YARN-10480. replace href tags with ng-href (#2426)
(cherry picked from commit 61f8c5767e)
2020-11-10 11:00:00 -08:00
Peter Bacsko
0361837c9e YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko. 2020-11-10 13:16:52 +01:00
Eric E Payne
1e22929885 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan)
(cherry picked from commit 55339c2bdd)
2020-11-05 22:08:51 +00:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne
(cherry picked from commit 8ee6bc2518)
2020-10-29 19:52:43 +00:00
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen
(cherry picked from commit bab5bf9743)
2020-10-28 10:33:41 -07:00
Eric Badger
8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics.
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780)
2020-10-15 18:09:00 +00:00
Jim Brennan
76e223a320 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein
(cherry picked from commit deb35a32ba)
2020-10-08 18:59:25 +00:00
Jim Brennan
c789e944b7 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne
(cherry picked from commit b361f29dda)
2020-10-06 18:19:53 +00:00
Adam Antal
3ae78e40bf YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan
(cherry picked from commit a1f7e760df)
2020-10-05 10:10:46 +02:00
Eric Yang
9176e8fe5d YARN-9809. Added node manager health status to resource manager registration call.
Contributed by Eric Badger via eyang

(cherry picked from commit e8dc862d38)
2020-09-28 16:41:53 +00:00
Hui Fei
ed19f63998
HADOOP-17277. Correct spelling errors for separator (#2322)
Contributed by Hui Fei.

(cherry picked from commit 474fa80bfb)
2020-09-23 15:39:51 +09:00
Jim Brennan
0ec21b9667 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T.
(cherry picked from commit 90894ea641)
2020-09-14 21:22:02 +00:00
Eric E Payne
87ff2f5597 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan).
(cherry picked from commit 9afec2ed17)
2020-09-11 16:19:09 +00:00
Brahma Reddy Battula
ea37a05d4b YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T.
(cherry picked from commit 43572fc7f8)
2020-09-09 17:09:35 +05:30
bibinchundatt
acb219b9c9 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan.
(cherry picked from commit 5d8600e80a)
2020-09-08 20:42:27 +00:00
Eric Badger
59e8508ce2 [YARN-10353] Log vcores used and cumulative cpu in containers monitor.
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d)
2020-09-08 15:37:35 +00:00
Adam Antal
5710005a19 YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan
(cherry picked from commit 34fe74da0e)
2020-09-07 11:46:30 +02:00
Sunil G
ea1c048a61 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke.
(cherry picked from commit 82ec28f442)
2020-08-19 11:56:40 +05:30
Jim Brennan
392a62278a YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger
(cherry picked from commit 4a7deae478)
2020-08-17 16:55:35 +00:00
hemanthboyina
5ededc0cba YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T.
(cherry picked from commit 86bbd38c8d)
2020-08-13 23:47:50 +05:30
Jonathan Hung
706d14c41d YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne
(cherry picked from commit 64753addba)
2020-08-07 17:43:16 -07:00
bibinchundatt
0c0ea4979e YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah
(cherry picked from commit 81da221c75)
2020-08-07 09:01:53 +05:30
Eric Badger
723f1f61d6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource.
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925e)
2020-08-05 18:56:56 +00:00
Brahma Reddy Battula
643ff4881d YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T.
(cherry picked from commit eac558380f)
2020-08-03 12:55:31 +05:30
Eric E Payne
0cb1b178ec YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan.
(cherry picked from commit e0c9653166)
2020-07-30 16:39:56 +00:00
Jonathan Hung
d3c7f7cdbd YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne
(cherry picked from commit 3eaf62726f)
2020-07-28 13:43:44 -07:00
Eric Badger
7ec692aa83 YARN-4771. Some containers can be skipped during log aggregation after NM
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
2020-07-24 22:46:57 +00:00
Ayush Saxena
e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Ahmed Hussein
9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
2020-07-15 11:40:13 -05:00
Ahmed Hussein
5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
2020-07-15 09:57:36 -05:00
Eric Badger
41bcef9486 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by
Jim Brennan

(cherry picked from commit 48f90115b5)
2020-07-13 23:12:18 +00:00
Eric E Payne
7044a007b3 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan)
(cherry picked from commit 0427100b75)
2020-07-13 19:02:40 +00:00
Brahma Reddy Battula
7b175739a9 YARN-10341. Yarn Service Container Completed event doesn't get processed. Contributed by Bilwa S T.
(cherry picked from commit dfe60392c9)
2020-07-09 12:36:21 +05:30
Akira Ajisaka
0aa2d7d506
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126)
(cherry picked from commit 10d218934c)
2020-07-09 15:07:44 +09:00
Akira Ajisaka
20df70a895
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein.
(cherry picked from commit 639acb6d89)
2020-07-06 16:09:37 +09:00
Szilard Nemeth
439c51425e YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 17:42:45 +02:00