big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,426 Commits 2 Branches 2 Tags 264 MiB
f90a703e48
Commit Graph

2958 Commits

Author SHA1 Message Date
gavin.wang
f5345b881d
YARN-11560. Fix NPE bug when multi-node enabled with schedule asynchronously (#6021). Contributed by wangzhongwei. 2024-09-23 13:09:15 +05:30
Arjun Mohnot
d8ca2dbe34
YARN-11730. Mark unreported nodes as LOST on RM Startup/HA failover (#7049) Contributed by Arjun Mohnot. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-09-23 10:46:20 +08:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)" 
This reverts commit fa9bb0d1ac.
 2024-08-29 14:42:03 +05:30
PJ Fanning
59dba6e1bd
HADOOP-19134. Use StringBuilder instead of StringBuffer. (#6692). Contributed by PJ Fanning 2024-08-18 21:29:12 +05:30
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953) 
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
 2024-08-15 16:44:54 +01:00
Yang Jiandan
167d4c8447
YARN-11699. Diagnostics lacks userlimit info when user capacity has reached its maximum limit (#6849) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-06-01 06:18:28 +08:00
刘斌
6c08e8e2aa
HADOOP-19156. ZooKeeper based state stores use different ZK address configs. (#6767). Contributed by liu bin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-05-29 20:44:36 +08:00
Tamas Domok
ecf665c6fa
YARN-11191. Fix potentional deadlock in GlobalScheduler refreshQueues (#6732) 2024-04-24 14:58:50 +02:00
Tamas Domok
a386ac1f56
YARN-11684. Fix general contract violation in PriorityQueueComparator. (#6725) Contributed by Tamas Domok. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-04-19 08:37:05 +08:00
PJ Fanning
f7d1ec2d9e
HADOOP-19077. Remove use of javax.ws.rs.core.HttpHeaders (#6554). Contributed by PJ Fanning 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-04-01 12:43:39 +05:30
PJ Fanning
97c5a6efba
HADOOP-19041. Use StandardCharsets in more places (#6449) 2024-03-28 23:17:18 -04:00
Junfan Zhang
f1f2abe641
YARN-11668. Fix RM crash for potential concurrent modification exception when updating node attributes (#6681) Contributed by Junfan Zhang. 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-29 09:45:16 +08:00
XiaobaoWu
a375ef8cfa
YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata (#6616) 
Co-authored-by: wuxiaobao <xbaowu@163.com>
 2024-03-21 03:12:14 -04:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-15 08:14:38 +08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
 2024-02-13 16:33:51 +00:00
Eremenko Valentin
141627778d
YARN-11362: Fix several typos in YARN codebase of misspelled resource (#6474) Contributed by EremenkoValentin. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:34:42 +08:00
Yang Jiandan
98259fb4da
YARN-11653. Add Totoal_Memory and Total_Vcores columns in Nodes page (#6501) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-31 07:07:54 +08:00
Yang Jiandan
dffb505725
YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (#6490) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-29 21:37:32 +08:00
Peter Szucs
4046751939
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#6499) 2024-01-26 14:42:59 +01:00
Tamas Domok
7272a3d960
YARN-11641. Fix getCapacityConfigurationTypeForQueues for absolute mode with zero resource. (#6435) 2024-01-25 13:58:00 +01:00
slfan1989
6b80b1e60f
Revert "YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup" (#6497) 2024-01-25 11:44:15 +08:00
Peter Szucs
dd72e8fbc3
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#5332) 2024-01-24 15:51:07 +01:00
Tamas Domok
af5c8e70ea
YARN-11645. Order the queues by queuePath in json assert tests. (#6432) 2024-01-24 10:12:24 +01:00
Ferenc Erdelyi
8243da8cb0
YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2024-01-22 15:41:48 +01:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
Tamas Domok
55b9f87698
YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. (#6433) 2024-01-11 13:47:00 +01:00
gavin.wang
de9fde8224
YARN-11563. Fix typo in AbstractContainerAllocator from CSAssignemnt to CSAssignment (#6026). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:43:56 +05:30
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
slfan1989
0c10bab7bb
YARN-11483. [Federation] Router AdminCLI Supports Clean Finish Apps. (#6251) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-17 23:34:36 +08:00
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider (#6263) 2023-11-17 01:51:43 -08:00
slfan1989
90e9aa272e
YARN-11484. [Federation] Router Supports Yarn Client CLI Cmds. (#6132) 2023-11-07 18:38:35 -08:00
slfan1989
72d7b43a32
YARN-11548. [Federation] Router Supports Format FederationStateStore. (#6116) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 07:41:56 +08:00
Benjamin Teke
077263d9f3
YARN-11608. Fix QueueCapacityVectorInfo NPE when accessible labels config is used. (#6250) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-06 10:11:14 +01:00
Brian Goerlitz
4c04a6768c
YARN-11584. Safely fail on leaf queue with empty name (#6148) 2023-10-31 17:25:41 +01:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
Szilard Nemeth
2736f88561 YARN.11590. RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely. Contributed by Ferenc Erdelyi 2023-10-16 15:17:58 -04:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
slfan1989
f3a27f2b22
YARN-11579. Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data. (#6123) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-05 14:20:40 +08:00
Szilard Nemeth
d9cb76ac98 YARN-11468. Zookeeper SSL/TLS support. Contributed by Ferenc Erdelyi 2023-09-27 18:21:45 -04:00
Benjamin Teke
f51162d70b
YARN-11514. Extend SchedulerResponse with capacityVector (#5989) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-09-25 16:24:03 +02:00
Szilard Nemeth
13c5825c00
YARN-11573. Add config option to make container allocation prefer nodes without reserved containers (#6098) 2023-09-22 20:00:50 +02:00
PJ Fanning
56b928b86f
YARN-11498. Add exclusion for jettison everywhere jersey-json is loaded (#5786) 
All uses  of jersey-json in the yarn and other hadoop modules now
exclude the obsolete org.codehaus.jettison/jettison and so avoid
all security issues which can come from the library.

Contributed by PJ Fanning
 2023-09-13 18:10:24 +01:00
zhengchenyu
c5e9510b54
YARN-8980. Mapreduce application container start fail after AM restart. (#5975) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-09 09:50:53 +08:00
slfan1989
e1dde3bc23
YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#5944) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-06 22:08:59 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
Benjamin Teke
43c889636a
YARN-11535. Remove jackson-dataformat-yaml dependency. (#5970) 2023-08-22 16:42:49 +02:00