big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,364 Commits 2 Branches 2 Tags 264 MiB
1658a5140a
Commit Graph

4152 Commits

Author SHA1 Message Date
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 17:51:01 +00:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 23:34:35 +00:00
Akira Ajisaka
c96fbb9199
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9)
 2021-03-30 17:47:54 +09:00
Eric Badger
65bba8c3ed YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 
(cherry picked from commit 19e418c10d)
 2021-03-25 22:35:19 +00:00
Jim Brennan
78bddd0d9f YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 174f3a96b1)
 2021-03-23 18:23:50 +00:00
Eric Badger
0e37da334b YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 
(cherry picked from commit 181b9ddc51)
 2021-03-18 17:47:18 +00:00
Eric Badger
8c82e3507d YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 
(cherry picked from commit 38495af325)
 2021-03-18 17:47:10 +00:00
Eric Badger
cd417f17ae YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 
(cherry picked from commit 49f89f1d3d)
 2021-03-17 18:16:59 +00:00
Eric Badger
1022ed91ac YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.

(cherry picked from commit 7d3c8ef606)
 2021-03-15 20:01:24 +00:00
Eric Payne
f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:14:19 +00:00
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:20:37 +00:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Brahma Reddy Battula
f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit b2a565629d)
 2021-03-09 20:27:07 +05:30
Peter Bacsko
066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
Eric Badger
1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.

(cherry picked from commit 9d088639bf)
 2021-03-05 19:49:49 +00:00
Neil
0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 
(cherry picked from commit d615e2d3bd)
 2021-03-04 14:54:28 +05:30
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 19:56:07 +00:00
Jonathan Hung
be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan
db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f88)
 2021-02-25 16:48:46 +00:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fb)
 2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f)
 2021-02-18 20:32:32 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edf)
 2021-02-18 20:23:56 +05:30
Prabhu Joseph
72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0b)
 2021-02-18 14:25:16 +05:30
Prabhu Joseph
0c46ab51b5 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850)
 2021-02-18 13:59:50 +05:30
Jim Brennan
457c8404b9 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit 3fcc7a55b6)
 2021-02-16 16:47:46 +00:00
Masatake Iwasaki
4468378e4b YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 
(cherry picked from commit f9a073c6c1)
 2021-02-11 20:26:09 +00:00
Jim Brennan
d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af43)
 2021-02-05 17:37:04 +00:00
bibinchundatt
1520b84b36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 
(cherry picked from commit 8bc2dfbf36)
 2021-01-22 08:30:12 +05:30
Eric Badger
7fc624aabc YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba)
 2021-01-13 23:43:15 +00:00
Eric Payne
4518ece4c9 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit d8fadec944)
 2021-01-13 17:17:07 +00:00
Neil
cd5ee0014f YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 
(cherry picked from commit fa4cf91b57)
 2021-01-08 10:38:08 -08:00
Szilard Nemeth
f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
srinivasst
98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
 2021-01-08 10:57:37 +05:30
Eric Badger
1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
 2020-12-21 23:11:47 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Eric Payne
1fd6d81617 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit fa773a8326)
 2020-12-01 22:51:20 +00:00
Peter Bacsko
0361837c9e YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko. 2020-11-10 13:16:52 +01:00
Eric E Payne
1e22929885 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bdd)
 2020-11-05 22:08:51 +00:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518)
 2020-10-29 19:52:43 +00:00
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743)
 2020-10-28 10:33:41 -07:00
Eric Badger
8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780)
 2020-10-15 18:09:00 +00:00
Jim Brennan
76e223a320 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32ba)
 2020-10-08 18:59:25 +00:00
Jim Brennan
c789e944b7 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit b361f29dda)
 2020-10-06 18:19:53 +00:00
Adam Antal
3ae78e40bf YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760df)
 2020-10-05 10:10:46 +02:00
Eric Yang
9176e8fe5d YARN-9809. Added node manager health status to resource manager registration call. 
Contributed by Eric Badger via eyang

(cherry picked from commit e8dc862d38)
 2020-09-28 16:41:53 +00:00
Jim Brennan
0ec21b9667 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 90894ea641)
 2020-09-14 21:22:02 +00:00
Eric E Payne
87ff2f5597 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed17)
 2020-09-11 16:19:09 +00:00
Brahma Reddy Battula
ea37a05d4b YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 
(cherry picked from commit 43572fc7f8)
 2020-09-09 17:09:35 +05:30