big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,309 Commits 2 Branches 2 Tags 264 MiB
167d4c8447
Commit Graph

549 Commits

Author SHA1 Message Date
slfan1989
9f6c997662
YARN-11471. [Federation] FederationStateStoreFacade Cache Support Caffeine. (#6795) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-06-01 06:15:20 +08:00
刘斌
6c08e8e2aa
HADOOP-19156. ZooKeeper based state stores use different ZK address configs. (#6767). Contributed by liu bin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-05-29 20:44:36 +08:00
slfan1989
a1ae35e691
HADOOP-19135. Remove Jcache 1.0-alpha. (#6695) Contributed by Shilun Fan. 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-04-05 22:09:15 +08:00
slfan1989
5f3eb446f7
YARN-11663. [Federation] Add Cache Entity Nums Limit. (#6662) Contributed by Shilun Fan. 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-04-02 07:47:59 +08:00
PJ Fanning
06db6289cb
HADOOP-19024. Use bouncycastle jdk18 1.77 (#6410). Contributed 2024-03-30 19:58:12 +05:30
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411) 
Co-authored-by: slfan1989 <slfan1989@apache.org>
 2024-01-19 15:05:22 +08:00
slfan1989
7935eded5e
YARN-11627. [GPG] Improve GPGPolicyFacade#getPolicyManager. (#6332) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-15 08:36:11 +08:00
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
slfan1989
53c3ae1c89
YARN-11610. [Federation] Add WeightedHomePolicyManager. (#6256) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-19 10:04:46 +08:00
slfan1989
0c10bab7bb
YARN-11483. [Federation] Router AdminCLI Supports Clean Finish Apps. (#6251) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-17 23:34:36 +08:00
slfan1989
72d7b43a32
YARN-11548. [Federation] Router Supports Format FederationStateStore. (#6116) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 07:41:56 +08:00
slfan1989
80a22a736e
YARN-11500. [Addendum] Fix typos in hadoop-yarn-server-common#federation. (#6212) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-24 09:28:05 +08:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
slfan1989
b00d605832
YARN-9048. Add znode hierarchy in Federation ZK State Store. (#6016) 2023-10-09 14:06:41 -07:00
slfan1989
fe3984aa01
YARN-11580. YARN Router Web supports displaying information for Non-Federation. (#6127) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-03 18:21:52 +08:00
slfan1989
1d2afc5cf6
YARN-8862. [BackPort] [GPG] Add Yarn Registry cleanup in ApplicationCleaner. (#6083) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-29 07:15:53 +08:00
slfan1989
8538af4638
YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator. (#5934) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Co-authored-by: slfan1989 <slfan1989@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-14 21:28:49 +08:00
slfan1989
e1dde3bc23
YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#5944) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-06 22:08:59 +08:00
slfan1989
a6c2526c6c
YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf. (#5967) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-03 20:08:40 +08:00
slfan1989
ea87aa2f5b
YARN-11037. Add configurable logic to split resource request to the least loaded SC. (#5515) 2023-08-15 13:28:29 -07:00
slfan1989
df0381f236
YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues. (#5862) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-08-10 10:27:36 +08:00
slfan1989
5056c267e6
YARN-11508. [Minor] Improve UnmanagedAMPoolManager/UnmanagedApplicationManager Code (#5726) 2023-07-28 20:20:23 +08:00
slfan1989
23ecc32d3a
YARN-11525. [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues. (#5816) 2023-07-19 11:51:47 -07:00
slfan1989
84dd624768
YARN-11531. [Federation] Code cleanup for NodeManager#amrmproxy. (#5841) 2023-07-19 11:50:38 -07:00
slfan1989
5c02f21f2e
YARN-11510. [Federation] Fix NodeManager#TestFederationInterceptor Flaky Unit Test. (#5733) 2023-06-20 10:25:46 -07:00
slfan1989
3d2cddd235
YARN-11516. Improve FederationStateStoreFacade#existsApplicationHomeSubCluster/existsReservationHomeSubCluster Log Level. (#5755) 2023-06-20 10:23:51 -07:00
slfan1989
a4c3d48c31
YARN-11504. [Federation] YARN Federation Supports Non-HA mode. (#5722) 2023-06-12 15:28:13 -07:00
slfan1989
2794fe264b
YARN-8898. [Addendum] Improve NodeManager#TestFederationInterceptor Setup Code (#5732) 2023-06-12 15:21:24 -07:00
slfan1989
a409d52ef8
YARN-6648. BackPort [GPG] Add SubClusterCleaner in Global Policy Generator. (#5676) 2023-06-12 15:18:43 -07:00
Dongjoon Hyun
fb16e00da0
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun. 
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-11 11:38:13 +05:30
slfan1989
fd3c3ae068
YARN-11500. Fix typos in hadoop-yarn-server-common#federation. (#5702) 2023-06-05 15:49:36 -07:00
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c.
 2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
slfan1989
b977065cc4
YARN-11478. [Federation] SQLFederationStateStore Support Store ApplicationSubmitData. (#5663) 2023-05-24 11:43:20 -07:00
slfan1989
bba663038d
YARN-8898. Fix FederationInterceptor#allocate to set application priority in allocateResponse. (#5645) 2023-05-18 11:57:38 -07:00
slfan1989
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00
slfan1989
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. (#5363) 2023-05-09 16:17:23 -07:00
slfan1989
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. (#5403) 2023-05-08 10:13:09 -07:00
slfan1989
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616) 2023-05-05 13:43:40 -07:00
slfan1989
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. (#5609) 2023-05-05 13:41:38 -07:00
slfan1989
c1d10f3872
YARN-9049. Add application submit data to state store. (#5606) 2023-05-03 09:19:54 -07:00
slfan1989
87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. (#5598) 2023-05-01 16:01:50 -07:00
slfan1989
5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. (#5570) 
Co-authored-by: slfan1989 <louj1988@@>
 2023-04-28 14:11:13 -07:00
slfan1989
a716459cdf
YARN-11437. [Federation] SQLFederationStateStore Support Version. (#5589) 2023-04-25 14:09:45 -07:00
slfan1989
635521db4c
YARN-11438. [Federation] ZookeeperFederationStateStore Support Version. (#5537) 2023-04-18 09:05:52 -07:00
slfan1989
69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
slfan1989
eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00