big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,036 Commits 2 Branches 2 Tags 264 MiB
f96361fd54
Commit Graph

4792 Commits

Author SHA1 Message Date
slfan1989
bd93b2007c
YARN-11619. [Federation] Router CLI Supports List SubClusters. (#6304) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-07 21:11:11 +08:00
slfan1989
931fa49363
YARN-11613. [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. (#6295) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-05 23:32:03 +08:00
slfan1989
8745857c3b
YARN-11561. [Federation] GPG Supports Format PolicyStateStore. (#6300) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-03 18:36:20 +08:00
slfan1989
478c4ced5a
YARN-11620. [Federation] Improve FederationClientInterceptor To Return Partial Results of subClusters. (#6289) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-29 07:11:35 +08:00
Benjamin Teke
d72cdf7205
YARN-11611. Remove json-io to 4.14.1 due to CVE-2023-34610 (#6257) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-27 15:39:09 +01:00
slfan1989
8a610e616f
YARN-11614. [Federation] Add Federation PolicyManager Validation Rules. (#6271) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-26 16:01:35 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
slfan1989
616e381c9f
YARN-11577. Improve FederationInterceptorREST Method Result. (#6190) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-20 20:55:25 +08:00
slfan1989
53c3ae1c89
YARN-11610. [Federation] Add WeightedHomePolicyManager. (#6256) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-19 10:04:46 +08:00
slfan1989
0c10bab7bb
YARN-11483. [Federation] Router AdminCLI Supports Clean Finish Apps. (#6251) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-17 23:34:36 +08:00
Istvan Fajth
7a55442297
HADOOP-18956. Zookeeper SSL/TLS support in ZKDelegationTokenSecretManager and ZKSignerSecretProvider (#6263) 2023-11-17 01:51:43 -08:00
slfan1989
90e9aa272e
YARN-11484. [Federation] Router Supports Yarn Client CLI Cmds. (#6132) 2023-11-07 18:38:35 -08:00
slfan1989
d9a6792ca9
YARN-11011. Make YARN Router throw Exception to client clearly. (#6211) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 10:07:54 +08:00
slfan1989
72d7b43a32
YARN-11548. [Federation] Router Supports Format FederationStateStore. (#6116) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-08 07:41:56 +08:00
Benjamin Teke
077263d9f3
YARN-11608. Fix QueueCapacityVectorInfo NPE when accessible labels config is used. (#6250) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-11-06 10:11:14 +01:00
WangYuanben
a79cd110a4
YARN-11609. [Addendum] [Federation] Router Supports DeregisterSubCluster. (#6254) YuanBen Wang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-05 07:50:41 +08:00
slfan1989
2b1378c99e
YARN-11583. [Addendum] Improve Node Link for YARN Federation Web Page. (#6232) 2023-11-01 11:51:38 -07:00
Brian Goerlitz
4c04a6768c
YARN-11584. Safely fail on leaf queue with empty name (#6148) 2023-10-31 17:25:41 +01:00
slfan1989
254dbab5a3
YARN-9013. [BackPort] [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner. (#6147) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-31 06:56:00 +08:00
PJ Fanning
a079f6261d
HADOOP-18917. Addendum. Fix deprecation issues after commons-io upgrade. (#6228). Contributed by PJ Fanning. 2023-10-30 09:35:02 +05:30
slfan1989
40e8300719
YARN-11592. Add timeout to GPGUtils#invokeRMWebService. (#6189) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-28 07:09:09 +08:00
slfan1989
652908519e
YARN-11588. [Federation] [Addendum] Fix uncleaned threads in yarn router thread pool executor. (#6222) 2023-10-26 13:39:06 -07:00
slfan1989
9c7e5b66fa
YARN-11576. Improve FederationInterceptorREST AuditLog. (#6117) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-24 09:36:06 +08:00
slfan1989
80a22a736e
YARN-11500. [Addendum] Fix typos in hadoop-yarn-server-common#federation. (#6212) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-24 09:28:05 +08:00
Zita Dombi
4c04818d3d
HADOOP-18919. Zookeeper SSL/TLS support in HDFS ZKFC (#6194) 2023-10-23 11:03:15 -07:00
Szilard Nemeth
2736f88561 YARN.11590. RM process stuck after calling confStore.format() when ZK SSL/TLS is enabled, as netty thread waits indefinitely. Contributed by Ferenc Erdelyi 2023-10-16 15:17:58 -04:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
jchanggg
bd28ba385a
YARN-11588. [Federation] Fix uncleaned threads in yarn router thread pool executor (#6159) Contributed by Jeffrey Chang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-12 19:13:44 +08:00
slfan1989
b00d605832
YARN-9048. Add znode hierarchy in Federation ZK State Store. (#6016) 2023-10-09 14:06:41 -07:00
slfan1989
42b32fbbdc
YARN-11583. Improve Node Link for YARN Federation Web Page. (#6145) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-08 08:20:11 +08:00
slfan1989
f3a27f2b22
YARN-11579. Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data. (#6123) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-05 14:20:40 +08:00
slfan1989
fe3984aa01
YARN-11580. YARN Router Web supports displaying information for Non-Federation. (#6127) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-03 18:21:52 +08:00
zhengchenyu
b8815fe68b
MAPREDUCE-7453. Revert HADOOP-18649. (#6102). Contributed by zhengchenyu. 
In container-log4j.properties, log4j.appender.{APPENDER}.MaxFileSize is set to ${yarn.app.container.log.filesize}, but yarn.app.container.log.filesize is 0 in default. So log is missing. This log is always rolling and only show the latest log.
 2023-10-01 17:25:32 +05:30
slfan1989
5f47f091a2
YARN-11537. [Addendum][Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#6121) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-29 16:59:01 +08:00
slfan1989
1d2afc5cf6
YARN-8862. [BackPort] [GPG] Add Yarn Registry cleanup in ApplicationCleaner. (#6083) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-29 07:15:53 +08:00
Szilard Nemeth
2d871fab78 MAPREDUCE-7456. Extend add-opens flag to container launch commands on JDK17 nodes. Contributed by Peter Szucs 2023-09-27 22:33:45 -04:00
Masatake Iwasaki
0c153fe465
YARN-11558. Fix dependency convergence error on hbase2 profile. (#6017) 2023-09-28 10:17:29 +09:00
Szilard Nemeth
d9cb76ac98 YARN-11468. Zookeeper SSL/TLS support. Contributed by Ferenc Erdelyi 2023-09-27 18:21:45 -04:00
slfan1989
3de66f5c40
YARN-11547. [Federation] Router Supports Remove individual application records from FederationStateStore. (#6055) 2023-09-25 13:52:57 -07:00
Benjamin Teke
f51162d70b
YARN-11514. Extend SchedulerResponse with capacityVector (#5989) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-09-25 16:24:03 +02:00
slfan1989
bf9975a1b3
YARN-9586. Need more doc for yarn.federation.policy-manager-params when LoadBasedRouterPolicy is used. (#6085) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-25 13:23:02 +08:00
Szilard Nemeth
13c5825c00
YARN-11573. Add config option to make container allocation prefer nodes without reserved containers (#6098) 2023-09-22 20:00:50 +02:00
K0K0V0K
0780710f25
YARN-11567 - Aggregate container launch debug artifacts on error (#6053) 2023-09-22 15:09:17 +02:00
slfan1989
8538af4638
YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator. (#5934) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Co-authored-by: slfan1989 <slfan1989@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-14 21:28:49 +08:00
PJ Fanning
56b928b86f
YARN-11498. Add exclusion for jettison everywhere jersey-json is loaded (#5786) 
All uses  of jersey-json in the yarn and other hadoop modules now
exclude the obsolete org.codehaus.jettison/jettison and so avoid
all security issues which can come from the library.

Contributed by PJ Fanning
 2023-09-13 18:10:24 +01:00
slfan1989
9c8cdbe28e
YARN-11433. Router's main() should support generic options. (#6012) 2023-09-12 14:48:40 -07:00
slfan1989
475932c524
YARN-11562. [Federation] GPG Support Query Policies In Web. (#6023) 2023-09-12 14:47:37 -07:00
slfan1989
e4550e1aef
YARN-6476. Advanced Federation UI based on YARN UI v2. (#5920) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-11 13:28:31 +08:00
zhengchenyu
c5e9510b54
YARN-8980. Mapreduce application container start fail after AM restart. (#5975) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-09 09:50:53 +08:00
slfan1989
3ab983fbae
YARN-11434. [Router] UGI conf doesn't read user overridden configurations on Router startup. (#6011) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-08 07:52:54 +08:00