big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,665 Commits 2 Branches 2 Tags 264 MiB
e19c00925f
Commit Graph

4253 Commits

Author SHA1 Message Date
Peter Bacsko
e19c00925f YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu. 2021-03-05 13:18:06 +01:00
Peter Bacsko
6699198b54 YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. 2021-03-04 17:18:35 +01:00
Peter Bacsko
a85aeee876 YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. 2021-03-04 12:23:11 +01:00
Neil
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 2021-03-04 14:52:58 +05:30
Peter Bacsko
7e8040e6ad YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. 2021-03-03 17:44:30 +01:00
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
Peter Bacsko
947b50489d YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke. 2021-02-26 21:57:54 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Jonathan Hung
ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan
c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko
d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko
4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Szilard Nemeth
9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Jim Brennan
3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Szilard Nemeth
bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665) 
Amogh Rajesh Desai via Zhankun Tang
 2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan
4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda) 
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
 2021-01-26 08:24:29 -08:00
Wangda Tan
b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda) 
Change-Id: I670e5525619b320745254609c48e7e1afb084835
 2021-01-26 08:24:28 -08:00
Szilard Nemeth
f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth
8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko
17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00