big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,047 Commits 2 Branches 2 Tags 264 MiB
6b4987f09c
Commit Graph

4331 Commits

Author SHA1 Message Date
Jack Jiang
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Cyrus Jackson
cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications (#2674). Contributed by Cyrus Jackson 2021-03-19 15:49:05 +05:30
Eric Badger
3d4ea4b336 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-03-18 19:36:50 +00:00
Eric Badger
181b9ddc51 YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 2021-03-18 17:45:09 +00:00
Peter Bacsko
ce6bfd5718 YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu. 2021-03-18 13:40:16 +01:00
Peter Bacsko
38495af325 YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 2021-03-18 12:46:11 +01:00
Szilard Nemeth
a5745711dd YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak 2021-03-18 12:43:01 +01:00
Peter Bacsko
d7eeca4d0c YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu. 2021-03-18 11:49:16 +01:00
Eric Badger
49f89f1d3d YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 2021-03-17 18:11:37 +00:00
Peter Bacsko
3e58d5611d YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu. 2021-03-17 13:38:20 +01:00
Wilfred Spiegelenburg
f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it 
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.

Contributed by: Siddharth Ahuja
 2021-03-17 10:55:05 +11:00
Peter Bacsko
73394fabc7 YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu. 2021-03-16 16:09:12 +01:00
Peter Bacsko
b80588b688 YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu. 2021-03-16 15:23:27 +01:00
zhuqi
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. (#2760) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-16 13:50:29 +09:00
Eric Payne
aa4c17b9d7 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 2021-03-15 19:09:40 +00:00
Jim Brennan
5f067cf0f3 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 2021-03-12 17:17:31 +00:00
Szilard Nemeth
5db4c0bf70 YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak 2021-03-12 14:10:16 +01:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Brahma Reddy Battula
b2a565629d YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 2021-03-09 20:26:07 +05:30
Peter Bacsko
d5e035dbe1 YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-03-09 14:33:14 +01:00
Peter Bacsko
3851994cd6 Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu." 
This reverts commit 369f75b7a7.
 2021-03-09 14:32:02 +01:00
Peter Bacsko
369f75b7a7 YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu. 2021-03-09 14:28:23 +01:00
Peter Bacsko
20416bc62d YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth. 2021-03-09 11:34:37 +01:00
Peter Bacsko
e472ee2aa5 YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth. 2021-03-09 11:28:24 +01:00
Peter Bacsko
04cd3115ba YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu. 2021-03-08 12:29:44 +01:00
Peter Bacsko
c6b30a59dd YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth. 2021-03-05 23:13:35 +01:00
Eric Badger
9d088639bf YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.
 2021-03-05 19:46:40 +00:00
Peter Bacsko
e82e7c597a YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu. 2021-03-05 15:56:56 +01:00
Peter Bacsko
e19c00925f YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu. 2021-03-05 13:18:06 +01:00
Peter Bacsko
6699198b54 YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. 2021-03-04 17:18:35 +01:00
Peter Bacsko
a85aeee876 YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. 2021-03-04 12:23:11 +01:00
Neil
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 2021-03-04 14:52:58 +05:30
Peter Bacsko
7e8040e6ad YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. 2021-03-03 17:44:30 +01:00
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
Peter Bacsko
947b50489d YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke. 2021-02-26 21:57:54 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Jonathan Hung
ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan
c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko
d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko
4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Szilard Nemeth
9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Jim Brennan
3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Szilard Nemeth
bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt
61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Neil
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Eric Badger
4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.
 2020-12-21 23:10:31 +00:00
zhuqi
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. (#2563) 
Qi Zhu via Zhankun Tang
 2020-12-21 10:04:16 +08:00
Akira Ajisaka
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka 2020-12-16 17:26:15 +01:00
Szilard Nemeth
df35c7f519 YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak 2020-12-15 17:30:14 +01:00
adamantal
3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
adamantal
0a45bd034e YARN-10520. Deprecated the residual nested class for the LCEResourceHandler (#2527). Contributed by Wanqiang Ji 2020-12-09 12:55:24 +01:00
zhuqi
d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler (#2494) 
Contributed by Qi Zhu.
 2020-12-09 19:48:39 +08:00
Eric Badger
7d3c8ef606 YARN-10495. make the rpath of container-executor configurable. Contributed by 
angerszhu.
 2020-12-08 00:53:24 +00:00
adamantal
9969745343 YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy 2020-12-03 09:55:06 +01:00
Eric Payne
fa773a8326 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-01 22:06:47 +00:00
Ankit Kumar
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-11-25 13:00:09 +09:00