The work to make it production-ready continues. Some key items include
* More robust storage fault tolerance
* Security
* Support for off-cluster clients
* More complete and integrated web UI
* Better support for long-running apps
* Offline (time-based periodic) aggregation for flows, users, and queues for reporting and
analysis
* Timeline collectors as separate instances from node managers
* Clustering of the readers
* Migration and compatibility with v.1
#<a name="Deployment"></a>Deployment
###<a name="Configurations"></a>Configurations
New configuration parameters that are introduced with v.2 are marked bold.
#### Basic configuration
| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the `TimelineClient` library used by applications will post entities and events to the Timeline server. Defaults to `false`. |
| `yarn.timeline-service.version` | Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and must bring up the timeline service v.1.5 (and nothing else). On the client side, if the client uses the same version of timeline service, it must succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. Defaults to `1.0f`. |
| **`yarn.timeline-service.writer.class`** | The class for the backend storage writer. Defaults to a filesystem storage writer, therefore it must be overridden. |
| **`yarn.timeline-service.reader.class`** | The class for the backend storage reader. Defaults to a filesystem storage reader, therefore it must be overridden. |
| **`yarn.system-metrics-publisher.enabled`** | The setting that controls whether yarn system metrics is published on the Timeline service or not by RM And NM. Defaults to `false`. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |
#### Advanced configuration
| Configuration Property | Description |
|:---- |:---- |
| `yarn.timeline-service.hostname` | The hostname of the Timeline service web application. Defaults to `0.0.0.0` |
| `yarn.timeline-service.address` | Address for the Timeline server to start the RPC server. Defaults to `${yarn.timeline-service.hostname}:10200`. |
| `yarn.timeline-service.webapp.address` | The http address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8188`. |
| `yarn.timeline-service.webapp.https.address` | The https address of the Timeline service web application. Defaults to `${yarn.timeline-service.hostname}:8190`. |
| **`yarn.timeline-service.writer.flush-interval-seconds`** | The setting that controls how often the timeline collector flushes the timeline writer. Defaults to `60`. |
| **`yarn.timeline-service.app-collector.linger-period.ms`** | Time period till which the application collector will be alive in NM, after the application master container finishes. Defaults to `1000` (1 second). |
| **`yarn.timeline-service.timeline-client.number-of-async-entities-to-merge`** | Time line V2 client tries to merge these many number of async entities (if available) and then call the REST ATS V2 API to submit. Defaults to `10`. |
| **`yarn.timeline-service.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). |
### <a name="Enabling_Timeline_Service_v2"></a>Enabling the Timeline Service v.2
#### Preparing Apache HBase cluster for storage
The first part is to set up or pick an Apache HBase cluster to use as the storage cluster. Once
you have an HBase cluster ready to use for this purpose, perform the following steps.
First, add the timeline service jar to the HBase classpath in all HBase machines in the cluster. It
is needed for the coprocessor as well as the schema creator. For example,
# <a name="Timeline_Service_REST_API_v2"></a>Timeline Service v.2 REST API
Querying the Timeline Service v.2 is currently only supported via REST API; there is no API
client implemented in the YARN libraries.
The v.2 REST API is implemented at under the path, `/ws/v2/timeline/` on the Timeline Service web
service.
Here is an informal description of the API.
### Root path
GET /ws/v2/timeline/
Returns a JSON object describing the service instance and version information.
{
"About":"Timeline Reader API",
"timeline-service-version":"3.0.0-SNAPSHOT",
"timeline-service-build-version":"3.0.0-SNAPSHOT from fb0acd08e6f0b030d82eeb7cbfa5404376313e60 by sjlee source checksum be6cba0e42417d53be16459e1685e7",
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means apps with
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all apps which have the specified
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`isRelatedTo` - If specified, matched applications must be related to or not related to given entities associated with a entity type. isRelatedTo is
represented in the same form as relatesTo.
1.`infofilters` - If specified, matched applications must have exact matches to the given info key and must be either equal or not equal to
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`conffilters` - If specified, matched applications must have exact matches to the given config name and must be either equal or not equal
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
1.`metricfilters` - If specified, matched applications must have exact matches to the given metric and satisfy the specified relation with the
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means apps with
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all apps which have the specified
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`isRelatedTo` - If specified, matched applications must be related to or not related to given entities associated with a entity type. isRelatedTo is
represented in the same form as relatesTo.
1.`infofilters` - If specified, matched applications must have exact matches to the given info key and must be either equal or not equal to
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`conffilters` - If specified, matched applications must have exact matches to the given config name and must be either equal or not equal
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
1.`metricfilters` - If specified, matched applications must have exact matches to the given metric and satisfy the specified relation with the
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
1. If successful, a HTTP 200 (OK) response is returned.
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
## <a name="REST_API_LIST_APP"></a>Query app
With this API, you can query a single YARN application identified by cluster and application ID. If REST endpoint without cluster name
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
flow context information based on cluster and app id.
### HTTP request:
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}
or
GET /ws/v2/timeline/apps/{app id}
### Query Parameters Supported:
1.`userid` - If specified, only applications belonging to this user will be returned. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowname` - Only applications belonging to this flowname will be returned. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowrunid` - Only applications belonging to this flow run id will be returned. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
metricsToRetrieve can be an expression of the form :<br/>
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for the
simple expression. Alternatively, expressions can be of the form :<br/>
With this API, you can query generic entities identified by cluster ID, application ID and per-framework entity type. If REST endpoint without cluster name
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
flow context information based on cluster and app id. If number of matching entities are more than the limit, the most recent entities up to the limit
will be returned. This endpoint can be used to query containers, application attempts or any other generic entity which clients put into the backend.
For instance, we can query containers by specifying entity type as `YARN_CONTAINER` and application attempts by specifying entity type as `YARN_APPLICATION_ATTEMPT`.
If none of the entities match the predicates, an empty list will be returned.
### HTTP request:
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type}
or
GET /ws/v2/timeline/apps/{app id}/entities/{entity type}
### Query Parameters Supported:
1.`userid` - If specified, only entities belonging to this user will be returned. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowname` - If specified, only entities belonging to this flowname will be returned. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowrunid` - If specified, only entities belonging to this flow run id will be returned. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`limit` - If specified, defines the number of entities to return. The maximum possible value for limit is maximum value of Long. If it is not specified
or has a value less than 0, then limit will be considered as 100.
1.`createdTimeStart` - If specified, then only entities created after this timestamp are returned.
1.`createdTimeEnd` - If specified, then only entities created before this timestamp are returned.
1.`relatesTo` - If specified, matched entities must relate to or not relate to given entities associated with a entity type.
relatesto is represented as an expression of the form :<br/>
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means entities with
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all entities which have the specified
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`isRelatedTo` - If specified, matched entities must be related to or not related to given entities associated with a entity type. isRelatedTo is
represented in the same form as relatesTo.
1.`infofilters` - If specified, matched entities must have exact matches to the given info key and must be either equal or not equal to
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
1.`conffilters` - If specified, matched entities must have exact matches to the given config name and must be either equal or not equal
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
1.`metricfilters` - If specified, matched entities must have exact matches to the given metric and satisfy the specified relation with the
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
With this API, you can query a specific generic entity identified by cluster ID, application ID, per-framework entity type and entity ID. If REST endpoint without
cluster name is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
flow context information based on cluster and app id. This endpoint can be used to query a single container, application attempt or any other generic entity which
clients put into the backend. For instance, we can query a specific YARN container by specifying entity type as `YARN_CONTAINER` and giving entity ID as container ID.
Similarly, application attempt can be queried by specifying entity type as `YARN_APPLICATION_ATTEMPT` and entity ID being the application attempt ID.
### HTTP request:
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type}/{entity id}
or
GET /ws/v2/timeline/apps/{app id}/entities/{entity type}/{entity id}
### Query Parameters Supported:
1.`userid` - If specified, entity must belong to this user. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowname` - If specified, entity must belong to this flow name. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`flowrunid` - If specified, entity must belong to this flow run id. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
1.`metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
metricsToRetrieve can be an expression of the form :<br/>
