big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,450 Commits 2 Branches 2 Tags 264 MiB
864b710787
Commit Graph

6016 Commits

Author SHA1 Message Date
Peter Bacsko
864b710787 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-19 10:31:57 +02:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
lujiefsi
137e20cc9b
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 13:53:27 +09:00
Aryan Gupta
5f147112cf
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 
(cherry picked from commit bbbfa7d415)
 2021-05-13 12:16:20 +08:00
Aryan Gupta
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
 2021-05-13 12:16:17 +08:00
Peter Bacsko
051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Borislav Iordanov
f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Szilard Nemeth
3d715c2e4c
YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 
(cherry picked from commit eacbe07b56)
 2021-05-07 14:32:17 +09:00
Szilard Nemeth
3303aa5947
YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 
(cherry picked from commit 8620984b8d)
 2021-05-07 14:26:58 +09:00
Akira Ajisaka
217655269a YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc1)
 2021-05-06 23:47:24 +09:00
Wei-Chiu Chuang
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029)
 2021-05-02 11:12:37 +09:00
Eric Badger
da2ebfa8a9 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 
(cherry picked from commit 803ac4b1a0)
 2021-04-29 17:52:03 +00:00
Wei-Chiu Chuang
df29aaaf80
YARN-10752. Shaded guava not found when compiling with profile hbase2.0. (#2957) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ic9943d18d4086b075ceed7924dcfa566653a16cb
 2021-04-27 00:41:42 +09:00
Eric Badger
003deeeecf YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 22:14:57 +00:00
Jim Brennan
3b1cde2f42 YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 
(cherry picked from commit 1cbe35946b)
 2021-04-23 21:38:02 +00:00
Eric Badger
1960924d07 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7)
 2021-04-20 17:46:05 +00:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 
(cherry picked from commit 31477a224f)
 2021-04-16 20:21:01 +00:00
dependabot[bot]
d0b02a23f3 Bump y18n 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit 1a90fc4b50068865359809cd437e45595c397195)
 2021-04-11 01:31:27 +09:00
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 17:51:01 +00:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 23:34:35 +00:00
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
Akira Ajisaka
c96fbb9199
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9)
 2021-03-30 17:47:54 +09:00
Brahma Reddy Battula
5358313f97 YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 413a4c3c05)
 2021-03-30 13:52:35 +05:30
Brahma Reddy Battula
bac1326e4e YARN-10439. addendum fix for shaded guva. 
(cherry picked from commit 6577bf1891)
 2021-03-30 13:49:23 +05:30
Brahma Reddy Battula
616a41ee32 YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit b61f52ec56)
 2021-03-30 09:52:28 +05:30
Brahma Reddy Battula
5181b2004b YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit d0dcfc405c)
 2021-03-30 09:46:52 +05:30
Brahma Reddy Battula
e9d8f16a70 YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 2d62dced4b)
 2021-03-30 09:40:06 +05:30
Eric Badger
65bba8c3ed YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 
(cherry picked from commit 19e418c10d)
 2021-03-25 22:35:19 +00:00
Jim Brennan
78bddd0d9f YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 174f3a96b1)
 2021-03-23 18:23:50 +00:00
Eric Badger
0e37da334b YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 
(cherry picked from commit 181b9ddc51)
 2021-03-18 17:47:18 +00:00
Eric Badger
8c82e3507d YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 
(cherry picked from commit 38495af325)
 2021-03-18 17:47:10 +00:00
Eric Badger
cd417f17ae YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 
(cherry picked from commit 49f89f1d3d)
 2021-03-17 18:16:59 +00:00
Eric Badger
1022ed91ac YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.

(cherry picked from commit 7d3c8ef606)
 2021-03-15 20:01:24 +00:00
Eric Payne
f5810ea83c YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:14:19 +00:00
Jim Brennan
2204bafbfb [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:20:37 +00:00
Akira Ajisaka
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-11 14:57:03 +09:00
Brahma Reddy Battula
f12293fba2 YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit b2a565629d)
 2021-03-09 20:27:07 +05:30
Peter Bacsko
eb05d49349 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 12:36:08 +01:00
Peter Bacsko
066f89af01 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 11:42:59 +01:00
Eric Badger
1a33878279 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.

(cherry picked from commit 9d088639bf)
 2021-03-05 19:49:49 +00:00
Neil
0396a721e3 YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 
(cherry picked from commit d615e2d3bd)
 2021-03-04 14:54:28 +05:30
Ahmed Hussein
792329fde9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a06)
 2021-02-26 19:56:07 +00:00
Eric Badger
e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:07:59 +00:00
Jonathan Hung
be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan
db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f88)
 2021-02-25 16:48:46 +00:00
Eric Badger
305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 18:59:45 +00:00
Eric Badger
27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:25:25 +00:00
Inigo Goiri
8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fb)
 2021-02-18 20:42:26 +05:30
Inigo Goiri
97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f)
 2021-02-18 20:32:32 +05:30
bibinchundatt
a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edf)
 2021-02-18 20:23:56 +05:30