big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,930 Commits 2 Branches 2 Tags 264 MiB
9bc159f4ac
Commit Graph

6564 Commits

Author SHA1 Message Date
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
jchanggg
bd28ba385a
YARN-11588. [Federation] Fix uncleaned threads in yarn router thread pool executor (#6159) Contributed by Jeffrey Chang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-12 19:13:44 +08:00
slfan1989
b00d605832
YARN-9048. Add znode hierarchy in Federation ZK State Store. (#6016) 2023-10-09 14:06:41 -07:00
slfan1989
42b32fbbdc
YARN-11583. Improve Node Link for YARN Federation Web Page. (#6145) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-08 08:20:11 +08:00
slfan1989
f3a27f2b22
YARN-11579. Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data. (#6123) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-05 14:20:40 +08:00
slfan1989
fe3984aa01
YARN-11580. YARN Router Web supports displaying information for Non-Federation. (#6127) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-10-03 18:21:52 +08:00
Tamas Domok
a04a9e107b
YARN-11578. Cache fs supports chmod in LogAggregationFileController. (#6120) 2023-10-02 15:20:47 +02:00
zhengchenyu
b8815fe68b
MAPREDUCE-7453. Revert HADOOP-18649. (#6102). Contributed by zhengchenyu. 
In container-log4j.properties, log4j.appender.{APPENDER}.MaxFileSize is set to ${yarn.app.container.log.filesize}, but yarn.app.container.log.filesize is 0 in default. So log is missing. This log is always rolling and only show the latest log.
 2023-10-01 17:25:32 +05:30
slfan1989
5f47f091a2
YARN-11537. [Addendum][Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#6121) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-29 16:59:01 +08:00
slfan1989
1d2afc5cf6
YARN-8862. [BackPort] [GPG] Add Yarn Registry cleanup in ApplicationCleaner. (#6083) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-29 07:15:53 +08:00
Szilard Nemeth
2d871fab78 MAPREDUCE-7456. Extend add-opens flag to container launch commands on JDK17 nodes. Contributed by Peter Szucs 2023-09-27 22:33:45 -04:00
Masatake Iwasaki
0c153fe465
YARN-11558. Fix dependency convergence error on hbase2 profile. (#6017) 2023-09-28 10:17:29 +09:00
Szilard Nemeth
d9cb76ac98 YARN-11468. Zookeeper SSL/TLS support. Contributed by Ferenc Erdelyi 2023-09-27 18:21:45 -04:00
Tamas Domok
f232eec490
YARN-11522. Update the documentation with the YARN-11000 changes. (#5870) 2023-09-27 16:43:28 +02:00
slfan1989
3de66f5c40
YARN-11547. [Federation] Router Supports Remove individual application records from FederationStateStore. (#6055) 2023-09-25 13:52:57 -07:00
Benjamin Teke
f51162d70b
YARN-11514. Extend SchedulerResponse with capacityVector (#5989) 
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
 2023-09-25 16:24:03 +02:00
slfan1989
bf9975a1b3
YARN-9586. Need more doc for yarn.federation.policy-manager-params when LoadBasedRouterPolicy is used. (#6085) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-25 13:23:02 +08:00
Szilard Nemeth
13c5825c00
YARN-11573. Add config option to make container allocation prefer nodes without reserved containers (#6098) 2023-09-22 20:00:50 +02:00
K0K0V0K
0780710f25
YARN-11567 - Aggregate container launch debug artifacts on error (#6053) 2023-09-22 15:09:17 +02:00
slfan1989
42b8e6faa7
YARN-11570. Add YARN_GLOBALPOLICYGENERATOR_HEAPSIZE to yarn-env for GPG. (#6086) 2023-09-20 17:11:59 -07:00
slfan1989
8538af4638
YARN-7599. [BackPort][GPG] ApplicationCleaner in Global Policy Generator. (#5934) Contributed by Botong Huang, Shilun Fan. 
Co-authored-by: Botong Huang <botong@apache.org>
Co-authored-by: slfan1989 <slfan1989@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-14 21:28:49 +08:00
PJ Fanning
56b928b86f
YARN-11498. Add exclusion for jettison everywhere jersey-json is loaded (#5786) 
All uses  of jersey-json in the yarn and other hadoop modules now
exclude the obsolete org.codehaus.jettison/jettison and so avoid
all security issues which can come from the library.

Contributed by PJ Fanning
 2023-09-13 18:10:24 +01:00
slfan1989
9c8cdbe28e
YARN-11433. Router's main() should support generic options. (#6012) 2023-09-12 14:48:40 -07:00
slfan1989
475932c524
YARN-11562. [Federation] GPG Support Query Policies In Web. (#6023) 2023-09-12 14:47:37 -07:00
slfan1989
e4550e1aef
YARN-6476. Advanced Federation UI based on YARN UI v2. (#5920) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-11 13:28:31 +08:00
zhengchenyu
c5e9510b54
YARN-8980. Mapreduce application container start fail after AM restart. (#5975) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-09 09:50:53 +08:00
zhengchenyu
bf605c8acc
YARN-11564. Fix wrong config in yarn-default.xml (#6030) Contributed by Chenyu Zheng. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-08 15:23:26 +08:00
slfan1989
3ab983fbae
YARN-11434. [Router] UGI conf doesn't read user overridden configurations on Router startup. (#6011) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-08 07:52:54 +08:00
slfan1989
e1dde3bc23
YARN-11537. [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. (#5944) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-06 22:08:59 +08:00
slfan1989
a6c2526c6c
YARN-11435. [Router] FederationStateStoreFacade is not reinitialized with Router conf. (#5967) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-03 20:08:40 +08:00
slfan1989
d5334fa761
YARN-6537. Running RM tests against the Router. (#5957) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-03 15:38:21 +08:00
slfan1989
b6d06c8928
YARN-11554. Fix TestRMFailover#testWebAppProxyInStandAloneMode Failed. (#5971) 2023-08-31 10:46:33 -07:00
Yang Jiandan
2f739450be
YARN-11552. Timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store (#5978) Contributed by Jiandan Yang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-01 00:14:46 +08:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf98.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
WangYuanben
39eaf8d14e
YARN-11556. Let Federation.md more standardized. (#5983) 2023-08-29 12:16:10 +08:00
Benjamin Teke
43c889636a
YARN-11535. Remove jackson-dataformat-yaml dependency. (#5970) 2023-08-22 16:42:49 +02:00
WangYuanben
7368226332
YARN-11553. Change the time unit of scCleanerIntervalMs in Router. (#5965) 2023-08-20 10:12:21 +08:00
slfan1989
ff1570acfa
YARN-10218. [GPG] Support HTTPS in GPG. (#5945) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-08-19 10:32:06 +08:00
zhengchenyu
136111314d
YARN-11154. Make router support proxy server. (#5946) 2023-08-19 10:29:26 +08:00
Benjamin Teke
ebc32fbcaf
YARN-11551. RM format-conf-store should delete all the content of ZKConfigStore (#5958) 2023-08-18 12:01:56 +02:00
slfan1989
ea87aa2f5b
YARN-11037. Add configurable logic to split resource request to the least loaded SC. (#5515) 2023-08-15 13:28:29 -07:00
slfan1989
8d95c588d2
YARN-8934. [BackPort][GPG] Add JvmMetricsInfo and pause monitor. (#5929) Contributed by Bilwa S T. 2023-08-12 09:15:34 +08:00
zhengchenyu
59f3a16819
YARN-11153. Make proxy server support YARN federation. (#4314) 2023-08-11 13:02:58 -07:00
Benjamin Teke
85b3ea6f50
MAPREDUCE-7449: Add add-opens flag to container launch commands on JDK17 nodes (#5935) 2023-08-10 22:47:17 +02:00
slfan1989
df0381f236
YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues. (#5862) Contributed by Shilun Fan. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-08-10 10:27:36 +08:00
slfan1989
001d353cc5
YARN-7708. BackPort [GPG] Load based policy generator. (#5902) Contributed by Young Chen. 2023-08-05 16:33:35 +08:00
slfan1989
6d3bcaa674
YARN-7402. BackPort [GPG] Fix potential connection leak in GPGUtils. (#5901) 2023-08-05 16:29:21 +08:00
susheel-gupta
8eb58630ec
YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder (#5320) 
Co-authored-by: Susheel Gupta <38013283+susheelgupta7@users.noreply.github.com>
 2023-08-04 10:32:55 +02:00
Peter Szucs
6e6349c793
YARN-11545. Fixed FS2CS ACL conversion when all users are allowed. (#5910) 2023-08-03 16:29:41 +02:00