YARN-11571. [GPG] Add Information About YARN GPG in Federation.md (#6158) Contributed by Shilun Fan.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
This commit is contained in:
slfan1989 2023-10-14 10:00:28 +08:00 committed by GitHub
parent 8963b25ab3
commit 00f8cdcb0f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 89 additions and 7 deletions

View File

@ -384,7 +384,7 @@ Optional:
|`yarn.federation.state-store.heartbeat-interval-secs` | `60` | The rate at which RMs report their membership to the federation to the central state-store. |
###ON ROUTER:
### ON ROUTER:
These are extra configurations that should appear in the **conf/yarn-site.xml** at each Router.
@ -465,7 +465,88 @@ If we want to use JCache, we can configure `yarn.federation.cache.class` to `org
This is a Cache implemented based on the Guava framework.
If we want to use it, we can configure `yarn.federation.cache.class` to `org.apache.hadoop.yarn.server.federation.cache.FederationGuavaCache`.
###ON NMs:
### ON GPG:
GlobalPolicyGenerator, abbreviated as "GPG," is used for the automatic generation of global policies for subClusters.
These are extra configurations that should appear in the **conf/yarn-site.xml** for GPG. We allow only one GPG.
Optional:
| Property | Example | Description |
|:------------------------------------------------------------------|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `yarn.federation.gpg.scheduled.executor.threads` | `10` | The number of threads to use for the GPG scheduled executor service. default is 10. |
| `yarn.federation.gpg.subcluster.cleaner.interval-ms` | `-1` | The interval at which the subcluster cleaner runs, -1 means disabled |
| `yarn.federation.gpg.subcluster.heartbeat.expiration-ms` | `30m` | The expiration time for a subcluster heartbeat, default is 30 minutes. |
| `yarn.federation.gpg.application.cleaner.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.DefaultApplicationCleaner` | The application cleaner class to use. |
| `yarn.federation.gpg.application.cleaner.interval-ms` | `-1` | The interval at which the application cleaner runs, -1 means disabled |
| `yarn.federation.gpg.application.cleaner.contact.router.spec` | `3,10,600000` | Should have three values separated by comma: minimal success retries, maximum total retry, retry interval (ms). |
| `yarn.federation.gpg.policy.generator.interval` | `1h` | The interval at which the policy generator runs, default is one hour. |
| `yarn.federation.gpg.policy.generator.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.NoOpGlobalPolicy` | The configured policy generator class, runs NoOpGlobalPolicy by default. |
| `yarn.federation.gpg.policy.generator.readonly` | `false` | Whether or not the policy generator is running in read only (won't modify policies), default is false.` |
| `yarn.federation.gpg.policy.generator.blacklist` | | Which sub-clusters the policy generator should blacklist. |
| `yarn.federation.gpg.policy.generator.load-based.pending.minimum` | `100` | The minimum number of pending applications in the subCluster. |
| `yarn.federation.gpg.policy.generator.load-based.pending.maximum` | `1000` | The maximum number of pending applications in the subCluster. |
| `yarn.federation.gpg.policy.generator.load-based.weight.minimum` | `0` | If a subCluster has a very high load, we will assign this value to the subCluster. The default value is 0, which means that we no longer assign appliaction to this subCluster. |
| `yarn.federation.gpg.policy.generator.load-based.edit.maximum` | `3` | This value represents the number of subClusters we want to calculate. default is 3. |
| `yarn.federation.gpg.policy.generator.load-based.scaling` | `LINEAR` | We provide 4 calculation methods: NONE, LINEAR, QUADRATIC, LOG. |
| `yarn.federation.gpg.webapp.address` | `0.0.0.0:8069` | The address of the GPG web application. |
| `yarn.federation.gpg.webapp.https.address` | `0.0.0.0:8070` | The https address of the GPG web application. |
- yarn.federation.gpg.application.cleaner.contact.router.spec
Specifications on how (many times) to contact Router for apps. We need to
do this because Router might return partial application list because some
sub-cluster RM is not responsive (e.g. failing over). Should have three values separated by comma: minimal success retries,
maximum total retry, retry interval (ms).
- yarn.federation.gpg.policy.generator.load-based.scaling
Note, this calculation method is when the number of Pending Applications in
the subCluster is less than yarn.federation.gpg.policy.generator.load-based.pending.maximum.
maxPendingVal = `yarn.federation.gpg.policy.generator.load-based.pending.maximum` -
`yarn.federation.gpg.policy.generator.load-based.pending.minimum`
curPendingVal = `Pending Applications in the subCluster` -
`yarn.federation.gpg.policy.generator.load-based.pending.minimum`
No calculation is required, and the weight is 1 at this time.
- LINEAR:
For linear computation,
we will use (maxPendingVal - curPendingVal) / (maxPendingVal).
- QUADRATIC:
Calculated using quadratic, We will calculate quadratic for maxPendingVal, curPendingVal,
then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal).
- LOG(LOGARITHM):
Calculated using logarithm, We will calculate logarithm for maxPendingVal, curPendingVal,
then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal).
LINEAR is used by default.
Security:
Kerberos supported in GPG.
| Property | Example | Description |
|:--------------------------------------------------|:--------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `yarn.federation.gpg.keytab.file` | | The keytab file used by GPG to login as its service principal. The principal name is configured with 'yarn.federation.gpg.kerberos.principal.hostname'. |
| `yarn.federation.gpg.kerberos.principal` | | The GPG service principal. This is typically set to GPG/_HOST@REALM.TLD. GPG will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on GPG in setup. |
| `yarn.federation.gpg.kerberos.principal.hostname` | | Optional. The hostname for the GPG containing this configuration file. Will be different for each machine. Defaults to current hostname. |
Enabling CORS support:
To enable cross-origin support (CORS) for the Yarn Router, please set the following configuration parameters:
| Property | Example | Description |
|---------------------------------------------------|---------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
| `hadoop.http.filter.initializers` | `org.apache.hadoop.security.HttpCrossOriginFilterInitializer` | Optional. Set the filter to HttpCrossOriginFilterInitializer, Configure this parameter in core-site.xml. |
| `yarn.federation.gpg.webapp.cross-origin.enabled` | `true` | Optional. Enable/disable CORS filter.Configure this parameter in yarn-site.xml. |
### ON NMs:
These are extra configurations that should appear in the **conf/yarn-site.xml** at each NodeManager.

View File

@ -36,12 +36,13 @@ Please note that, If you run YARN daemons locally in your machine for test purpo
you need the following configurations added to `yarn-site.xml` to enable cross
origin (CORS) support.
| Configuration Property | Value | Description |
|:---- |:---- |:---- |
| Configuration Property | Value | Description |
|:---- |:---- |:-----------------------------------------|
| `yarn.timeline-service.http-cross-origin.enabled` | true | Enable CORS support for Timeline Server |
| `yarn.resourcemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Resource Manager |
| `yarn.nodemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Node Manager |
| `yarn.router.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn Router |
| `yarn.resourcemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Resource Manager |
| `yarn.nodemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Node Manager |
| `yarn.router.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn Router |
| `yarn.federation.gpg.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn GPG |
Also please ensure that CORS related configurations are enabled in `core-site.xml`.
Kindly refer [here](../../hadoop-project-dist/hadoop-common/HttpAuthentication.html)