big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,215 Commits 2 Branches 2 Tags 264 MiB
aabf31f6fa
Commit Graph

6634 Commits

Author SHA1 Message Date
PJ Fanning
97c5a6efba
HADOOP-19041. Use StandardCharsets in more places (#6449) 2024-03-28 23:17:18 -04:00
Junfan Zhang
f1f2abe641
YARN-11668. Fix RM crash for potential concurrent modification exception when updating node attributes (#6681) Contributed by Junfan Zhang. 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-29 09:45:16 +08:00
XiaobaoWu
a375ef8cfa
YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata (#6616) 
Co-authored-by: wuxiaobao <xbaowu@163.com>
 2024-03-21 03:12:14 -04:00
Peter Szucs
a957cd5049
YARN-5305. Allow log aggregation to discard expired delegation tokens (#6625) 2024-03-20 15:33:10 +01:00
slfan1989
ff3f2255d2
HADOOP-19112. Hadoop 3.4.0 release wrap-up. (#6640) Contributed by Shilun Fan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-19 20:08:03 +08:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator (#6623) Contributed by Junfan Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-15 08:14:38 +08:00
PJ Fanning
fc166d3aec
HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fanning. 2024-03-07 15:09:01 +05:30
Steve Loughran
dae871e3e0
YARN-11657. Remove protobuf-2.5 from hadoop-yarn-api module (#6575) (#6580) 
The import of protobuf-java-2.5 in the hadoop-yarn-api module
is downgraded from "compile" to "provided"

This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar

It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar

Contributed by Steve Loughran
 2024-03-05 11:01:14 +00:00
Steve Loughran
a0ce2170db
HADOOP-19084. Prune hadoop-common transitive dependencies (#6574) (#6582) 
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
    
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging

This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.

Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.

Contributed by Steve Loughran
 2024-03-01 12:51:13 +00:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>


Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607). 

Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
 2024-02-13 16:33:51 +00:00
slfan1989
8011b21c52
HADOOP-19069. Use hadoop-thirdparty 1.2.0. (#6533) Contributed by Shilun Fan 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-08 19:18:04 +08:00
Eremenko Valentin
141627778d
YARN-11362: Fix several typos in YARN codebase of misspelled resource (#6474) Contributed by EremenkoValentin. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:34:42 +08:00
Yang Jiandan
98259fb4da
YARN-11653. Add Totoal_Memory and Total_Vcores columns in Nodes page (#6501) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-31 07:07:54 +08:00
Yang Jiandan
dffb505725
YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (#6490) Contributed by Jiandan Yang. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-29 21:37:32 +08:00
Peter Szucs
4046751939
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#6499) 2024-01-26 14:42:59 +01:00
Tamas Domok
7272a3d960
YARN-11641. Fix getCapacityConfigurationTypeForQueues for absolute mode with zero resource. (#6435) 2024-01-25 13:58:00 +01:00
slfan1989
6b80b1e60f
Revert "YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup" (#6497) 2024-01-25 11:44:15 +08:00
Peter Szucs
dd72e8fbc3
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#5332) 2024-01-24 15:51:07 +01:00
Tamas Domok
af5c8e70ea
YARN-11645. Order the queues by queuePath in json assert tests. (#6432) 2024-01-24 10:12:24 +01:00
Ferenc Erdelyi
8243da8cb0
YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2024-01-22 15:41:48 +01:00
Susheel Gupta
d0df0689b4
YARN-11607: TestTimelineAuthFilterForV2 fails intermittently (#6459) Contributed by Susheel Gupta. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-20 07:42:08 +08:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
PJ Fanning
04e447cfa7
YARN-11647. use StandardCharsets.UTF_8 (#6447) Contributed by PJ Fanning. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-18 13:53:18 +08:00
slfan1989
827e33601e
YARN-11638. [GPG] GPG Support CLI. (#6396) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-16 21:49:51 +08:00
slfan1989
6ebce65ae8
YARN-11634. [Addendum] Speed-up TestTimelineClient. (#6419) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-15 08:44:17 +01:00
Tamas Domok
55b9f87698
YARN-11646. Do not ignore zero memory capacity config in QueueCapacityConfigParser. (#6433) 2024-01-11 13:47:00 +01:00
slfan1989
bc159b5a87
YARN-10125. [Federation] Kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM. (#6363) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-11 20:01:59 +08:00
slfan1989
64beecb7cb
YARN-11631. [GPG] Add GPGWebServices. (#6354) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-06 17:50:20 +08:00
slfan1989
60033fd581
YARN-11642. Fix Flaky Test TestTimelineAuthFilterForV2#testPutTimelineEntities. (#6417) Contributed by Shilun Fan. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-06 16:26:01 +08:00
slfan1989
556fbcf025
YARN-11632. [Doc] Add allow-partial-result description to Yarn Federation documentation. (#6340) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-03 07:17:37 +08:00
K0K0V0K
5dd1977800
YARN-11634. Speed-up TestTimelineClient (#6371) 
The TimelineConnector.class has a hardcoded 1 minute connection time out, what makes the TestTimelineClient to a long running test (~15:30 min).
Decrease the timeout to 10ms will speed up the test run (~56 sec).
 2023-12-20 12:10:07 +01:00
gavin.wang
de9fde8224
YARN-11563. Fix typo in AbstractContainerAllocator from CSAssignemnt to CSAssignment (#6026). Contributed by wangzhongwei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-19 22:43:56 +05:30
slfan1989
3caabb2957
YARN-11629. [GPG] Improve GPGOverviewBlock Infomation. (#6341) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 12:36:57 +08:00
slfan1989
7935eded5e
YARN-11627. [GPG] Improve GPGPolicyFacade#getPolicyManager. (#6332) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 08:36:11 +08:00
Peter Szucs
16edb4ac96
YARN-11630. Passing admin Java options to container localizers (#6349) 2023-12-14 16:52:57 +01:00
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
K0K0V0K
5d1f889432
YARN-11420. Stabilize TestNMClient (#5317) 2023-12-05 11:32:42 +01:00
Susheel Gupta
78d5fe24d6
YARN-11621: Fix intermittently failing unit test: TestAMRMProxy.testAMRMProxyTokenRenewal (#6310) 2023-12-05 11:26:46 +01:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
PJ Fanning
3cb3dfafe5
HADOOP-18924. Upgrade to grpc 1.53.0 due to CVEs (#6161). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-01 09:53:47 +05:30
slfan1989
478c4ced5a
YARN-11620. [Federation] Improve FederationClientInterceptor To Return Partial Results of subClusters. (#6289) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-29 07:11:35 +08:00
Benjamin Teke
d72cdf7205
YARN-11611. Remove json-io to 4.14.1 due to CVE-2023-34610 (#6257) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-27 15:39:09 +01:00
slfan1989
8a610e616f
YARN-11614. [Federation] Add Federation PolicyManager Validation Rules. (#6271) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-26 16:01:35 +08:00
slfan1989
abd550cff4
YARN-11485. [Federation] Router Supports Yarn Admin CLI Cmds. (#6265) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-26 09:21:22 +08:00
Junfan Zhang
88e760bfc8
YARN-11616. Fast fail for NodeConstraintParser when having multi attribute kvs (#6282) 2023-11-24 14:34:35 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
slfan1989
616e381c9f
YARN-11577. Improve FederationInterceptorREST Method Result. (#6190) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-20 20:55:25 +08:00
slfan1989
53c3ae1c89
YARN-11610. [Federation] Add WeightedHomePolicyManager. (#6256) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-19 10:04:46 +08:00
slfan1989
0c10bab7bb
YARN-11483. [Federation] Router AdminCLI Supports Clean Finish Apps. (#6251) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-17 23:34:36 +08:00