big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,428 Commits 2 Branches 2 Tags 264 MiB
21ec686be3
Commit Graph

2960 Commits

Author SHA1 Message Date
Syed Shameerur Rahman
21ec686be3
YARN-11702: Fix Yarn over allocating containers (#6990) Contributed by Syed Shameerur Rahman. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-09-25 12:10:15 +08:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
 2024-09-24 16:50:22 +01:00
gavin.wang
f5345b881d
YARN-11560. Fix NPE bug when multi-node enabled with schedule asynchronously (#6021). Contributed by wangzhongwei. 2024-09-23 13:09:15 +05:30
Arjun Mohnot
d8ca2dbe34
YARN-11730. Mark unreported nodes as LOST on RM Startup/HA failover (#7049) Contributed by Arjun Mohnot. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-09-23 10:46:20 +08:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)" 
This reverts commit fa9bb0d1ac.
 2024-08-29 14:42:03 +05:30
PJ Fanning
59dba6e1bd
HADOOP-19134. Use StringBuilder instead of StringBuffer. (#6692). Contributed by PJ Fanning 2024-08-18 21:29:12 +05:30
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953) 
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
 2024-08-15 16:44:54 +01:00
Yang Jiandan
167d4c8447
YARN-11699. Diagnostics lacks userlimit info when user capacity has reached its maximum limit (#6849) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-06-01 06:18:28 +08:00
刘斌
6c08e8e2aa
HADOOP-19156. ZooKeeper based state stores use different ZK address configs. (#6767). Contributed by liu bin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-05-29 20:44:36 +08:00
Tamas Domok
ecf665c6fa
YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues (#6732) 2024-04-24 14:58:50 +02:00
Tamas Domok
a386ac1f56
YARN-11684. Fix general contract violation in PriorityQueueComparator. (#6725) Contributed by Tamas Domok. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-04-19 08:37:05 +08:00
PJ Fanning
f7d1ec2d9e
HADOOP-19077. Remove use of javax.ws.rs.core.HttpHeaders (#6554). Contributed by PJ Fanning 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-04-01 12:43:39 +05:30
PJ Fanning
97c5a6efba
HADOOP-19041. Use StandardCharsets in more places (#6449) 2024-03-28 23:17:18 -04:00
Junfan Zhang
f1f2abe641
YARN-11668. Fix RM crash for potential concurrent modification exception when updating node attributes (#6681) Contributed by Junfan Zhang. 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-29 09:45:16 +08:00
XiaobaoWu
a375ef8cfa
YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata (#6616) 
Co-authored-by: wuxiaobao <xbaowu@163.com>
 2024-03-21 03:12:14 -04:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-15 08:14:38 +08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
 2024-02-13 16:33:51 +00:00
Eremenko Valentin
141627778d
YARN-11362: Fix several typos in YARN codebase of misspelled resource (#6474) Contributed by EremenkoValentin. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:34:42 +08:00
Yang Jiandan
98259fb4da
YARN-11653. Add Totoal_Memory and Total_Vcores columns in Nodes page (#6501) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-31 07:07:54 +08:00
Yang Jiandan
dffb505725
YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (#6490) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-29 21:37:32 +08:00
Peter Szucs
4046751939
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#6499) 2024-01-26 14:42:59 +01:00
Tamas Domok
7272a3d960
YARN-11641. Fix getCapacityConfigurationTypeForQueues for absolute mode with zero resource. (#6435) 2024-01-25 13:58:00 +01:00
slfan1989
6b80b1e60f
Revert "YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup" (#6497) 2024-01-25 11:44:15 +08:00
Peter Szucs
dd72e8fbc3
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#5332) 2024-01-24 15:51:07 +01:00
Tamas Domok
af5c8e70ea
YARN-11645. Order the queues by queuePath in json assert tests. (#6432) 2024-01-24 10:12:24 +01:00
Ferenc Erdelyi
8243da8cb0
YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2024-01-22 15:41:48 +01:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
Tamas Domok
55b9f87698
YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. (#6433) 2024-01-11 13:47:00 +01:00
gavin.wang
de9fde8224
YARN-11563. Fix typo in AbstractContainerAllocator from CSAssignemnt to CSAssignment (#6026). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:43:56 +05:30
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
slfan1989
0c10bab7bb
YARN-11483. [Federation] Router AdminCLI Supports Clean Finish Apps. (#6251) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-17 23:34:36 +08:00
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider (#6263) 2023-11-17 01:51:43 -08:00
slfan1989
90e9aa272e
YARN-11484. [Federation] Router Supports Yarn Client CLI Cmds. (#6132) 2023-11-07 18:38:35 -08:00
slfan1989
72d7b43a32
YARN-11548. [Federation] Router Supports Format FederationStateStore. (#6116) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 07:41:56 +08:00
Benjamin Teke
077263d9f3
YARN-11608. Fix QueueCapacityVectorInfo NPE when accessible labels config is used. (#6250) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-06 10:11:14 +01:00
Brian Goerlitz
4c04a6768c
YARN-11584. Safely fail on leaf queue with empty name (#6148) 2023-10-31 17:25:41 +01:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
Szilard Nemeth
2736f88561 YARN.11590. RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely. Contributed by Ferenc Erdelyi 2023-10-16 15:17:58 -04:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
slfan1989
f3a27f2b22
YARN-11579. Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data. (#6123) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-05 14:20:40 +08:00
Szilard Nemeth
d9cb76ac98 YARN-11468. Zookeeper SSL/TLS support. Contributed by Ferenc Erdelyi 2023-09-27 18:21:45 -04:00
Benjamin Teke
f51162d70b
YARN-11514. Extend SchedulerResponse with capacityVector (#5989) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-09-25 16:24:03 +02:00
Szilard Nemeth
13c5825c00
YARN-11573. Add config option to make container allocation prefer nodes without reserved containers (#6098) 2023-09-22 20:00:50 +02:00
PJ Fanning
56b928b86f
YARN-11498. Add exclusion for jettison everywhere jersey-json is loaded (#5786) 
All uses  of jersey-json in the yarn and other hadoop modules now
exclude the obsolete org.codehaus.jettison/jettison and so avoid
all security issues which can come from the library.

Contributed by PJ Fanning
 2023-09-13 18:10:24 +01:00
zhengchenyu
c5e9510b54
YARN-8980. Mapreduce application container start fail after AM restart. (#5975) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-09 09:50:53 +08:00
slfan1989
e1dde3bc23
YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#5944) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-06 22:08:59 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00