big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

YARN-9804. Update ATSv2 document for latest feature supports.

Browse Source
This commit is contained in:
Rohith Sharma K S 2019-09-05 08:42:17 +05:30
parent 2f70b52a5b
commit 3db7184082
1 changed files with 15 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
View File

@ -73,8 +73,6 @@ The following diagram illustrates the design at a high level.
### <a name="Current_Status"></a>Current Status and Future Plans ### <a name="Current_Status"></a>Current Status and Future Plans
YARN Timeline Service v.2 is currently in alpha ("alpha 2"). It is a work in progress, and
many things can and will change rapidly.
A complete end-to-end flow of writes and reads is functional, with Apache HBase as the backend. A complete end-to-end flow of writes and reads is functional, with Apache HBase as the backend.
You should be able to start generating data. When enabled, all YARN-generic events are You should be able to start generating data. When enabled, all YARN-generic events are
@ -82,32 +80,31 @@ published as well as YARN system metrics such as CPU and memory. Furthermore, so
including Distributed Shell and MapReduce can write per-framework data to YARN Timeline Service including Distributed Shell and MapReduce can write per-framework data to YARN Timeline Service
v.2. v.2.
The basic mode of accessing data is via REST. Currently there is no support for command line The basic mode of accessing data is via REST. The REST API comes with a good number of useful and flexible query patterns (see below for
access. The REST API comes with a good number of useful and flexible query patterns (see below for more information). YARN Client has been integrated with ATSv2. This enables fetching application/attempt/container
more information). report from TimelineReader if details are not present in ResouceManager.
The collectors (writers) are currently embedded in the node managers as auxiliary services. The The collectors (writers) are currently embedded in the node managers as auxiliary services. The
resource manager also has its dedicated in-process collector. The reader is currently a single resource manager also has its dedicated in-process collector. The reader is currently a single
instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN
application (i.e. no off-cluster client). application (i.e. no off-cluster client).
Starting from alpha2, Timeline Service v.2 supports simple authorization in terms of a Kerberos Authentication is supported end to end. All communication to HBase can be kerberized. Refer [Security Configuration](#Security_Configuration) for configs.
configurable whitelist of users and groups who can read timeline data. Cluster admins are Support for simple authorization has been added in terms of a configurable whitelist of users and groups who can read timeline data. Cluster admins are
allowed by default to read timeline data. allowed by default to read timeline data.
When YARN Timeline Service v.2 is disabled, one can expect no functional or performance impact When YARN Timeline Service v.2 is disabled, one can expect no functional or performance impact
on any other existing functionality. on any other existing functionality.
The work to make it truly production-ready continues. Some key items include
* More robust storage fault tolerance Road map includes
* More robust storage fault tolerance.
* Support for off-cluster clients * Support for off-cluster clients
* Better support for long-running apps * Support for entity ACLs
* Support for ACLs
* Offline (time-based periodic) aggregation for flows, users, and queues for reporting and * Offline (time-based periodic) aggregation for flows, users, and queues for reporting and
analysis analysis
* Timeline collectors as separate instances from node managers
* Clustering of the readers
* Migration and compatibility with v.1 * Migration and compatibility with v.1
@ -143,7 +140,7 @@ New configuration parameters that are introduced with v.2 are marked bold.
| **`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). This should be set in the HBase cluster. | | **`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). This should be set in the HBase cluster. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. | | **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |
| **`yarn.nodemanager.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by NM. This configuration setting is for ATS V2. Defaults to `true`. | | **`yarn.nodemanager.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by NM. This configuration setting is for ATS V2. Defaults to `true`. |
#### Security Configuration #### <a name="Security_Configuration"></a>Security Configuration
Security can be enabled by setting `yarn.timeline-service.http-authentication.type` Security can be enabled by setting `yarn.timeline-service.http-authentication.type`
@ -163,6 +160,7 @@ to `kerberos`, after which the following configuration options are available:
| `yarn.timeline-service.delegation.token.max-lifetime` | Defaults to `604800000` (7 days). | | `yarn.timeline-service.delegation.token.max-lifetime` | Defaults to `604800000` (7 days). |
| `yarn.timeline-service.read.authentication.enabled` | Enables or disables authorization checks for reading timeline service v2 data. Default is `false` which is disabled. | | `yarn.timeline-service.read.authentication.enabled` | Enables or disables authorization checks for reading timeline service v2 data. Default is `false` which is disabled. |
| `yarn.timeline-service.read.allowed.users` | Comma separated list of user, followed by space, then comma separated list of groups. It will allow this list of users and groups to read the data and reject everyone else. Default value is set to none. If authorization is enabled, then this configuration is mandatory. | | `yarn.timeline-service.read.allowed.users` | Comma separated list of user, followed by space, then comma separated list of groups. It will allow this list of users and groups to read the data and reject everyone else. Default value is set to none. If authorization is enabled, then this configuration is mandatory. |
| `yarn.webapp.filter-entity-list-by-user` | Default is false. If set true and yarn.timeline-service.read.authentication.enabled is disabled, then listing of entities restricted to remote user entities only. It is YARN common configuration for listing APIs. Using this configuration TimelineReader authorize caller UGI with entity owner. If does not match, those entities will be removed from response.|
#### Enabling CORS support #### Enabling CORS support
To enable cross-origin support (CORS) for the Timeline Service v.2, please set the following configuration parameters: To enable cross-origin support (CORS) for the Timeline Service v.2, please set the following configuration parameters:
@ -408,13 +406,13 @@ To write MapReduce framework data to Timeline Service v.2, enable the following
</property> </property>
``` ```
### Upgrade from alpha1 to alpha2 ### Upgrade from alpha1 to GA
If you are currently running Timeline Service v2 alpha1 version, we recommend the following: If you are currently running Timeline Service v2 alpha1 version, we recommend the following:
- Clear existing data in tables (truncate tables) since the row key for AppToFlow has changed. - Clear existing data in tables (truncate tables) since the row key for AppToFlow has changed.
- The coprocessor is now a dynamically loaded table level coprocessor in alpha2. We - The coprocessor is now a dynamically loaded table level coprocessor in GA. We
recommend dropping the table, replacing the coprocessor jar on hdfs with the alpha2 one, recommend dropping the table, replacing the coprocessor jar on hdfs with the GA one,
restarting the Region servers and recreating the `flowrun` table. restarting the Region servers and recreating the `flowrun` table.
### <a name="Publishing_of_application_specific_data"></a> Publishing application specific data ### <a name="Publishing_of_application_specific_data"></a> Publishing application specific data