YARN-5052 Update timeline service v2 documentation to capture information about filters (Varun Saxena via Vrushali C)
This commit is contained in:
Vrushali Channapattan
Sangjin Lee
parent
892b193bd7
commit
32b033d57c
@ -128,6 +128,7 @@ REST API compatibility corresponds to both the request (URLs) and responses to e
|
|||||||
* [MR Application Master](../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html)
|
* [MR Application Master](../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html)
|
||||||
* [History Server](../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html)
|
* [History Server](../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html)
|
||||||
* [Timeline Server v1 REST API](../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html)
|
* [Timeline Server v1 REST API](../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html)
|
||||||
|
* [Timeline Service v2 REST API](../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html)
|
||||||
|
|
||||||
#### Policy
|
#### Policy
|
||||||
|
|
||||||
|
|||||||
@ -126,6 +126,7 @@
|
|||||||
<item name="Node Labels" href="hadoop-yarn/hadoop-yarn-site/NodeLabel.html"/>
|
<item name="Node Labels" href="hadoop-yarn/hadoop-yarn-site/NodeLabel.html"/>
|
||||||
<item name="Web Application Proxy" href="hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html"/>
|
<item name="Web Application Proxy" href="hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html"/>
|
||||||
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html"/>
|
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html"/>
|
||||||
|
<item name="Timeline Service V.2" href="hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html"/>
|
||||||
<item name="Writing YARN Applications" href="hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html"/>
|
<item name="Writing YARN Applications" href="hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html"/>
|
||||||
<item name="YARN Application Security" href="hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html"/>
|
<item name="YARN Application Security" href="hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html"/>
|
||||||
<item name="NodeManager" href="hadoop-yarn/hadoop-yarn-site/NodeManager.html"/>
|
<item name="NodeManager" href="hadoop-yarn/hadoop-yarn-site/NodeManager.html"/>
|
||||||
@ -141,6 +142,7 @@
|
|||||||
<item name="Resource Manager" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html"/>
|
<item name="Resource Manager" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html"/>
|
||||||
<item name="Node Manager" href="hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html"/>
|
<item name="Node Manager" href="hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html"/>
|
||||||
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1"/>
|
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1"/>
|
||||||
|
<item name="Timeline Service V.2" href="hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_REST_API_v2"/>
|
||||||
</menu>
|
</menu>
|
||||||
|
|
||||||
<menu name="Hadoop Compatible File Systems" inherit="top">
|
<menu name="Hadoop Compatible File Systems" inherit="top">
|
||||||
|
|||||||
@ -24,6 +24,14 @@ The YARN Timeline Service v.2
|
|||||||
* [Enabling the Timeline Service v.2](#Enabling_Timeline_Service_v2)
|
* [Enabling the Timeline Service v.2](#Enabling_Timeline_Service_v2)
|
||||||
* [Publishing of application specific data](#Publishing_of_application_specific_data)
|
* [Publishing of application specific data](#Publishing_of_application_specific_data)
|
||||||
* [Timeline Service v.2 REST API](#Timeline_Service_REST_API_v2)
|
* [Timeline Service v.2 REST API](#Timeline_Service_REST_API_v2)
|
||||||
|
* [Query Flows](#REST_API_LIST_FLOWS)
|
||||||
|
* [Query Flow Runs](#REST_API_LIST_FLOW_RUNS)
|
||||||
|
* [Query Flow Run](#REST_API_LIST_FLOW_RUN)
|
||||||
|
* [Query Apps for a Flow](#REST_API_LIST_FLOW_APPS)
|
||||||
|
* [Query Apps for a Flow Run](#REST_API_LIST_FLOWRUN_APPS)
|
||||||
|
* [Query App](#REST_API_LIST_APP)
|
||||||
|
* [Query Generic Entities](#REST_API_LIST_ENTITIES)
|
||||||
|
* [Query Generic Entity](#REST_API_LIST_ENTITY)
|
||||||
|
|
||||||
#<a name="Overview"></a>Overview
|
#<a name="Overview"></a>Overview
|
||||||
|
|
||||||
@ -78,11 +86,11 @@ The following diagram illustrates the design at a high level.
|
|||||||
### <a name="Current_Status"></a>Current Status and Future Plans
|
### <a name="Current_Status"></a>Current Status and Future Plans
|
||||||
|
|
||||||
YARN Timeline Service v.2 is currently in alpha. It is very much work in progress, and many things
|
YARN Timeline Service v.2 is currently in alpha. It is very much work in progress, and many things
|
||||||
can and will change rapidly. Users should enable Timeline Service v.2 only on a test or
|
can and will change rapidly. Users must enable Timeline Service v.2 only on a test or
|
||||||
experimental cluster to test the feature.
|
experimental cluster to test the feature.
|
||||||
|
|
||||||
A complete end-to-end flow of writes and reads should be functional, with Apache HBase as the
|
A complete end-to-end flow of writes and reads must be functional, with Apache HBase as the
|
||||||
backend. You should be able to start generating data. When enabled, all YARN-generic events are
|
backend. You must be able to start generating data. When enabled, all YARN-generic events are
|
||||||
published as well as YARN system metrics such as CPU and memory. Furthermore, some applications
|
published as well as YARN system metrics such as CPU and memory. Furthermore, some applications
|
||||||
including Distributed Shell and MapReduce write per-framework data to YARN Timeline Service v.2.
|
including Distributed Shell and MapReduce write per-framework data to YARN Timeline Service v.2.
|
||||||
|
|
||||||
@ -97,7 +105,7 @@ resource manager also has its dedicated in-process collector. The reader is curr
|
|||||||
instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN
|
instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN
|
||||||
application (i.e. no off-cluster client).
|
application (i.e. no off-cluster client).
|
||||||
|
|
||||||
When YARN Timeline Service v.2 is disabled, one should expect no functional or performance impact
|
When YARN Timeline Service v.2 is disabled, one must expect no functional or performance impact
|
||||||
on any other existing functionality.
|
on any other existing functionality.
|
||||||
|
|
||||||
The work to make it production-ready continues. Some key items include
|
The work to make it production-ready continues. Some key items include
|
||||||
@ -125,9 +133,9 @@ New configuration parameters that are introduced with v.2 are marked bold.
|
|||||||
| Configuration Property | Description |
|
| Configuration Property | Description |
|
||||||
|:---- |:---- |
|
|:---- |:---- |
|
||||||
| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the `TimelineClient` library used by applications will post entities and events to the Timeline server. Defaults to `false`. |
|
| `yarn.timeline-service.enabled` | Indicate to clients whether Timeline service is enabled or not. If enabled, the `TimelineClient` library used by applications will post entities and events to the Timeline server. Defaults to `false`. |
|
||||||
| `yarn.timeline-service.version` | Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and should bring up the timeline service v.1.5 (and nothing else). On the client side, if the client uses the same version of timeline service, it should succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. Defaults to `1.0f`. |
|
| `yarn.timeline-service.version` | Indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and must bring up the timeline service v.1.5 (and nothing else). On the client side, if the client uses the same version of timeline service, it must succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary. Defaults to `1.0f`. |
|
||||||
| **`yarn.timeline-service.writer.class`** | The class for the backend storage writer. Defaults to a filesystem storage writer, therefore it should be overridden. |
|
| **`yarn.timeline-service.writer.class`** | The class for the backend storage writer. Defaults to a filesystem storage writer, therefore it must be overridden. |
|
||||||
| **`yarn.timeline-service.reader.class`** | The class for the backend storage reader. Defaults to a filesystem storage reader, therefore it should be overridden. |
|
| **`yarn.timeline-service.reader.class`** | The class for the backend storage reader. Defaults to a filesystem storage reader, therefore it must be overridden. |
|
||||||
| **`yarn.system-metrics-publisher.enabled`** | The setting that controls whether yarn system metrics is published on the Timeline service or not by RM And NM. Defaults to `false`. |
|
| **`yarn.system-metrics-publisher.enabled`** | The setting that controls whether yarn system metrics is published on the Timeline service or not by RM And NM. Defaults to `false`. |
|
||||||
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |
|
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. |
|
||||||
|
|
||||||
@ -275,7 +283,7 @@ on `TimelineClient` suitable for writing to the Timeline Service v.2 are clearly
|
|||||||
they use the v.2 types as arguments.
|
they use the v.2 types as arguments.
|
||||||
|
|
||||||
Timeline Service v.2 `putEntities` methods come in 2 varieties: `putEntities` and
|
Timeline Service v.2 `putEntities` methods come in 2 varieties: `putEntities` and
|
||||||
`putEntitiesAsync`. The former is a blocking operation which should be used for writing more
|
`putEntitiesAsync`. The former is a blocking operation which must be used for writing more
|
||||||
critical data (e.g. lifecycle events). The latter is a non-blocking operation. Note that neither
|
critical data (e.g. lifecycle events). The latter is a non-blocking operation. Note that neither
|
||||||
has a return value.
|
has a return value.
|
||||||
|
|
||||||
@ -324,7 +332,7 @@ Service, while an off-cluster MapReduce job submitter cannot.
|
|||||||
|
|
||||||
You can create and publish your own entities, events, and metrics as with previous versions.
|
You can create and publish your own entities, events, and metrics as with previous versions.
|
||||||
|
|
||||||
Application frameworks should set the "flow context" whenever possible in order to take advantage
|
Application frameworks must set the "flow context" whenever possible in order to take advantage
|
||||||
of the flow support Timeline Service v.2 provides. The flow context consists of the following:
|
of the flow support Timeline Service v.2 provides. The flow context consists of the following:
|
||||||
|
|
||||||
* Flow name: a string that identifies the high-level flow (e.g. "distributed grep" or any
|
* Flow name: a string that identifies the high-level flow (e.g. "distributed grep" or any
|
||||||
@ -377,16 +385,40 @@ Returns a JSON object describing the service instance and version information.
|
|||||||
"hadoop-version-built-on":"2016-04-11T23:14Z"
|
"hadoop-version-built-on":"2016-04-11T23:14Z"
|
||||||
}
|
}
|
||||||
|
|
||||||
### Request Examples
|
The following shows the supported queries on the REST API.
|
||||||
|
|
||||||
The following shows some of the supported queries on the REST API. For example, to get the most
|
## <a name="REST_API_LIST_FLOWS"></a>Query Flows
|
||||||
recent flow activities,
|
|
||||||
|
|
||||||
HTTP request:
|
With Query Flows API, you can retrieve a list of active flows that had runs most recently.
|
||||||
|
If REST endpoint without cluster name is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken.
|
||||||
|
If none of the flows match the predicates, an empty list will be returned.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
GET /ws/v2/timeline/clusters/{cluster name}/flows/
|
GET /ws/v2/timeline/clusters/{cluster name}/flows/
|
||||||
|
|
||||||
Response:
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/flows/
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `limit` - If specified, defines the number of flows to return. The maximum possible value for limit
|
||||||
|
is maximum value of Long. If it is not specified or has a value less than 0, then limit will be
|
||||||
|
considered as 100.
|
||||||
|
1. `dateRange` - If specified is given as "[startdate]-[enddate]"(i.e. start and end date separated by
|
||||||
|
"-") or single date. Dates are interpreted in yyyyMMdd format and are assumed to be in GMT.
|
||||||
|
If a single date is specified, all flows active on that date are returned. If both startdate and enddate
|
||||||
|
is given, all flows active between start and end date will be returned. If only startdate is given, flows
|
||||||
|
active on and after startdate are returned. If only enddate is given, flows active on and before enddate
|
||||||
|
are returned.<br/>
|
||||||
|
_For example_ :<br/>
|
||||||
|
"daterange=20150711" returns flows active on 20150711.<br/>
|
||||||
|
"daterange=20150711-20150714" returns flows active between these 2 dates.<br/>
|
||||||
|
"daterange=20150711-" returns flows active on and after 20150711.<br/>
|
||||||
|
"daterange=-20150711" returns flows active on and before 20150711.<br/>
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
@ -439,15 +471,49 @@ Response:
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
It returns the flows that had runs (specific instances of the flows) most recently.
|
### Response Codes
|
||||||
|
|
||||||
You can drill further down to get the runs (specific instances) of a given flow.
|
1. If successful, a HTTP 200 (OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
HTTP request:
|
|
||||||
|
## <a name="REST_API_LIST_FLOW_RUNS"></a>Query Flow Runs
|
||||||
|
|
||||||
|
With Query Flow Runs API, you can drill further down to get the runs (specific instances) of a given flow.
|
||||||
|
This returns the most recent runs that belong to the given flow. If REST endpoint without cluster name is used,
|
||||||
|
cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken.
|
||||||
|
If none of the flow runs match the predicates, an empty list will be returned.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/users/{user name}/flows/{flow name}/runs/
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs/
|
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs/
|
||||||
|
|
||||||
Response:
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `limit` - If specified, defines the number of flows to return. The maximum possible value for limit
|
||||||
|
is maximum value of Long. If it is not specified or has a value less than 0, then limit will be
|
||||||
|
considered as 100.
|
||||||
|
1. `createdTimeStart` - If specified, then only flow runs started after this timestamp are returned.
|
||||||
|
1. `createdTimeEnd` - If specified, then only flow runs started before this timestamp are returned.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. For querying flow runs, only `ALL` or `METRICS` are valid fields.
|
||||||
|
Other fields will lead to HTTP 400 (Bad Request) response. If not specified, in response, id, type, createdtime and info fields
|
||||||
|
will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
@ -484,56 +550,291 @@ Response:
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
This returns the most recent runs that belong to the given flow.
|
### Response Codes
|
||||||
|
|
||||||
You can provide a `limit` query parameter to limit the number of entries that returned in a query.
|
1. If successful, a HTTP 200 (OK) response is returned.
|
||||||
If you want to limit the number of flow runs in the above query, you can do the following:
|
1. If any problem occurs in parsing request or if an invalid field is specified in fields query param, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
HTTP request:
|
|
||||||
|
|
||||||
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs?limit=1
|
## <a name="REST_API_LIST_FLOW_RUN"></a>Query Flow Run
|
||||||
|
|
||||||
Response:
|
With this API, you can query a specific flow run identified by cluster, user, flow name and run id.If REST endpoint without cluster name is
|
||||||
|
used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Metrics are returned by default while
|
||||||
|
querying individual flow run.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/users/{user name}/flows/{flow name}/runs/{run id}
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs/{run id}
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
|
{
|
||||||
|
"metrics": [
|
||||||
|
{
|
||||||
|
"type": "SINGLE_VALUE",
|
||||||
|
"id": "org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter:BYTES_READ",
|
||||||
|
"aggregationOp": "NOP",
|
||||||
|
"values": {
|
||||||
|
"1465246377261": 118
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "SINGLE_VALUE",
|
||||||
|
"id": "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter:BYTES_WRITTEN",
|
||||||
|
"aggregationOp": "NOP",
|
||||||
|
"values": {
|
||||||
|
"1465246377261": 97
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"events": [],
|
||||||
|
"id": "varun@QuasiMonteCarlo/1465246348599",
|
||||||
|
"type": "YARN_FLOW_RUN",
|
||||||
|
"createdtime": 1465246348599,
|
||||||
|
"isrelatedto": {},
|
||||||
|
"info": {
|
||||||
|
"UID":"yarn-cluster!varun!QuasiMonteCarlo!1465246348599",
|
||||||
|
"SYSTEM_INFO_FLOW_RUN_END_TIME":1465246378051,
|
||||||
|
"SYSTEM_INFO_FLOW_NAME":"QuasiMonteCarlo",
|
||||||
|
"SYSTEM_INFO_USER":"varun",
|
||||||
|
"SYSTEM_INFO_FLOW_RUN_ID":1465246348599
|
||||||
|
},
|
||||||
|
"relatesto": {}
|
||||||
|
}
|
||||||
|
|
||||||
|
### Response Codes
|
||||||
|
|
||||||
|
1. If successful, a HTTP 200(OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. If flow run for the given flow run id cannot be found, HTTP 404 (Not Found) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
|
## <a name="REST_API_LIST_FLOW_APPS"></a>Query Apps for a flow
|
||||||
|
|
||||||
|
With this API, you can query all the YARN applications that are part of a specific flow.If REST endpoint without cluster name
|
||||||
|
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. If number of matching applications are
|
||||||
|
more than the limit, the most recent apps up to the limit will be returned. If none of the apps match the predicates, an empty list will be returned.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/users/{user name}/flows/{flow name}/apps
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/apps
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `limit` - If specified, defines the number of applications to return. The maximum possible value for limit
|
||||||
|
is maximum value of Long. If it is not specified or has a value less than 0, then limit will be
|
||||||
|
considered as 100.
|
||||||
|
1. `createdTimeStart` - If specified, then only applications created after this timestamp are returned.
|
||||||
|
1. `createdTimeEnd` - If specified, then only applications created before this timestamp are returned.
|
||||||
|
1. `relatesTo` - If specified, matched applications must relate to or not relate to given entities associated with a entity type.
|
||||||
|
relatesto is represented as an expression of the form :<br/>
|
||||||
|
"(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...) <op> !(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...)".<br/>
|
||||||
|
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means apps with
|
||||||
|
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all apps which have the specified
|
||||||
|
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
|
||||||
|
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `isRelatedTo` - If specified, matched applications must be related to or not related to given entities associated with a entity type. isRelatedTo is
|
||||||
|
represented in the same form as relatesTo.
|
||||||
|
1. `infofilters` - If specified, matched applications must have exact matches to the given info key and must be either equal or not equal to
|
||||||
|
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
|
||||||
|
"(<key> <compareop> <value>) <op> (<key> <compareop> <value>)".<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
|
||||||
|
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `conffilters` - If specified, matched applications must have exact matches to the given config name and must be either equal or not equal
|
||||||
|
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
|
||||||
|
1. `metricfilters` - If specified, matched applications must have exact matches to the given metric and satisfy the specified relation with the
|
||||||
|
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
|
||||||
|
"(<metricid> <compareop> <metricvalue>) <op> (<metricid> <compareop> <metricvalue>)".<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne", "ene", "gt", "ge", "lt" and "le".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of metric is not required for a match, "ene" means not equals but existence of metric is
|
||||||
|
required, "gt" means greater than, "ge" means greater than or equals, "lt" means less than and "le" means less than or equals. We can combine
|
||||||
|
any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : metricfilters can be "(((metric1 eq 50) AND (metric2 gt 40)) OR (metric1 lt 20))".<br/>
|
||||||
|
This in essence is an expression equivalent to "(metric1 == 50 AND metric2 > 40) OR (metric1 < 20)"<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `eventfilters` - If specified, matched applications must contain or not contain the given events depending on the expression. eventfilters is
|
||||||
|
represented as an expression of the form :<br/>
|
||||||
|
"(<eventid>,<eventid>) <op> !(<eventid>,<eventid>,<eventid>)".<br/>
|
||||||
|
Here, "!" means none of the comma-separated list of events within the enclosed brackets proceeding "!" must exist for a match to occur.
|
||||||
|
If "!" is not specified, the specified events within the enclosed brackets must exist. op is a logical operator and can be either AND or OR.
|
||||||
|
We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : eventfilters can be "(((event1,event2) AND !(event4)) OR (event3,event7,event5))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for
|
||||||
|
the simple expression. Alternatively, expressions can be of the form:<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `confsToRetrieve` - If specified, defines which configs to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
confsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for
|
||||||
|
the simple expression. Alternatively, expressions can be of the form:<br/>
|
||||||
|
!(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If confsToRetrieve is specified, configs will be retrieved irrespective of whether `CONFIGS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. Possible values for fields can be `EVENTS`, `INFO`, `CONFIGS`, `METRICS`, `RELATES_TO`,
|
||||||
|
`IS_RELATED_TO` and `ALL`. All fields will be retrieved if `ALL` is specified. Multiple fields can be specified as a comma-separated list.
|
||||||
|
If fields is not specified, in response, app id, type (equivalent to YARN_APPLICATION), app createdtime and UID in info field will be returned.
|
||||||
|
1. `metricsLimit` - If specified, defines the number of metrics to return. Considered only if fields contains METRICS/ALL
|
||||||
|
or metricsToRetrieve is specified. Ignored otherwise. The maximum possible value for metricsLimit can be maximum value of
|
||||||
|
Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricsLimit will be
|
||||||
|
considered as 1 i.e. latest single value of metric(s) will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"metrics": [ ],
|
"metrics": [ ],
|
||||||
"events": [ ],
|
"events": [ ],
|
||||||
"id": "sjlee@ds-date/1460420587974",
|
"type": "YARN_APPLICATION",
|
||||||
"type": "YARN_FLOW_RUN",
|
"id": "application_1465246237936_0001",
|
||||||
"createdtime": 1460420587974,
|
"createdtime": 1465246348599,
|
||||||
"info": {
|
|
||||||
"UID": "test-cluster!sjlee!ds-date!1460420587974",
|
|
||||||
"SYSTEM_INFO_FLOW_RUN_ID": 1460420587974,
|
|
||||||
"SYSTEM_INFO_FLOW_NAME": "ds-date",
|
|
||||||
"SYSTEM_INFO_FLOW_RUN_END_TIME": 1460420595198,
|
|
||||||
"SYSTEM_INFO_USER": "sjlee"
|
|
||||||
},
|
|
||||||
"isrelatedto": { },
|
"isrelatedto": { },
|
||||||
|
"configs": { },
|
||||||
|
"info": {
|
||||||
|
"UID": "yarn-cluster!application_1465246237936_0001"
|
||||||
|
},
|
||||||
|
"relatesto": { }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"metrics": [ ],
|
||||||
|
"events": [ ],
|
||||||
|
"type": "YARN_APPLICATION",
|
||||||
|
"id": "application_1464983628730_0005",
|
||||||
|
"createdtime": 1465033881959,
|
||||||
|
"isrelatedto": { },
|
||||||
|
"configs": { },
|
||||||
|
"info": {
|
||||||
|
"UID": "yarn-cluster!application_1464983628730_0005"
|
||||||
|
},
|
||||||
"relatesto": { }
|
"relatesto": { }
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
Most queries in the v.2 REST API support the following query parameters:
|
### Response Codes
|
||||||
|
|
||||||
* `limit`
|
1. If successful, a HTTP 200 (OK) response is returned.
|
||||||
* `createdtimestart`
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
* `createdtimeend`
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
* `relatesto`
|
|
||||||
* `isrelatedto`
|
|
||||||
* `infofilters`
|
|
||||||
* `conffilters`
|
|
||||||
* `metricfilters`
|
|
||||||
* `eventfilters`
|
|
||||||
* `fields`
|
|
||||||
|
|
||||||
Given a flow run, you can query all the YARN applications that are part of that flow run:
|
## <a name="REST_API_LIST_FLOWRUN_APPS"></a>Query Apps for a flow run
|
||||||
|
|
||||||
HTTP request:
|
With this API, you can query all the YARN applications that are part of a specific flow run.If REST endpoint without cluster name
|
||||||
|
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. If number of matching applications are
|
||||||
|
more than the limit, the most recent apps up to the limit will be returned. If none of the apps match the predicates, an empty list will be returned.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/users/{user name}/flows/{flow name}/runs/{run id}/apps
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs/{run id}/apps/
|
GET /ws/v2/timeline/users/{user name}/flows/{flow name}/runs/{run id}/apps/
|
||||||
|
|
||||||
Response:
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `limit` - If specified, defines the number of applications to return. The maximum possible value for limit
|
||||||
|
is maximum value of Long. If it is not specified or has a value less than 0, then limit will be
|
||||||
|
considered as 100.
|
||||||
|
1. `createdTimeStart` - If specified, then only applications created after this timestamp are returned.
|
||||||
|
1. `createdTimeEnd` - If specified, then only applications created before this timestamp are returned.
|
||||||
|
1. `relatesTo` - If specified, matched applications must relate to or not relate to given entities associated with a entity type.
|
||||||
|
relatesto is represented as an expression of the form :<br/>
|
||||||
|
"(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...) <op> !(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...)".<br/>
|
||||||
|
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means apps with
|
||||||
|
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all apps which have the specified
|
||||||
|
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
|
||||||
|
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `isRelatedTo` - If specified, matched applications must be related to or not related to given entities associated with a entity type. isRelatedTo is
|
||||||
|
represented in the same form as relatesTo.
|
||||||
|
1. `infofilters` - If specified, matched applications must have exact matches to the given info key and must be either equal or not equal to
|
||||||
|
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
|
||||||
|
"(<key> <compareop> <value>) <op> (<key> <compareop> <value>)".<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
|
||||||
|
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `conffilters` - If specified, matched applications must have exact matches to the given config name and must be either equal or not equal
|
||||||
|
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
|
||||||
|
1. `metricfilters` - If specified, matched applications must have exact matches to the given metric and satisfy the specified relation with the
|
||||||
|
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
|
||||||
|
"(<metricid> <compareop> <metricvalue>) <op> (<metricid> <compareop> <metricvalue>)".<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne", "ene", "gt", "ge", "lt" and "le".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of metric is not required for a match, "ene" means not equals but existence of metric is
|
||||||
|
required, "gt" means greater than, "ge" means greater than or equals, "lt" means less than and "le" means less than or equals. We can combine
|
||||||
|
any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : metricfilters can be "(((metric1 eq 50) AND (metric2 gt 40)) OR (metric1 lt 20))".<br/>
|
||||||
|
This in essence is an expression equivalent to "(metric1 == 50 AND metric2 > 40) OR (metric1 < 20)"<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `eventfilters` - If specified, matched applications must contain or not contain the given events depending on the expression. eventfilters is
|
||||||
|
represented as an expression of the form :<br/>
|
||||||
|
"(<eventid>,<eventid>) <op> !(<eventid>,<eventid>,<eventid>)".<br/>
|
||||||
|
Here, "!" means none of the comma-separated list of events within the enclosed brackets proceeding "!" must exist for a match to occur.
|
||||||
|
If "!" is not specified, the specified events within the enclosed brackets must exist. op is a logical operator and can be either AND or OR.
|
||||||
|
We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : eventfilters can be "(((event1,event2) AND !(event4)) OR (event3,event7,event5))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `confsToRetrieve` - If specified, defines which configs to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
confsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If confsToRetrieve is specified, configs will be retrieved irrespective of whether `CONFIGS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. Possible values for fields can be `EVENTS`, `INFO`, `CONFIGS`, `METRICS`, `RELATES_TO`,
|
||||||
|
`IS_RELATED_TO` and `ALL`. All fields will be retrieved if `ALL` is specified. Multiple fields can be specified as a comma-separated list.
|
||||||
|
If fields is not specified, in response, app id, type (equivalent to YARN_APPLICATION), app createdtime and UID in info field will be returned.
|
||||||
|
1. `metricsLimit` - If specified, defines the number of metrics to return. Considered only if fields contains METRICS/ALL
|
||||||
|
or metricsToRetrieve is specified. Ignored otherwise. The maximum possible value for metricsLimit can be maximum value of
|
||||||
|
Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricsLimit will be
|
||||||
|
considered as 1 i.e. latest single value of metric(s) will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
@ -541,9 +842,9 @@ Response:
|
|||||||
"events": [],
|
"events": [],
|
||||||
"id": "application_1460419579913_0002",
|
"id": "application_1460419579913_0002",
|
||||||
"type": "YARN_APPLICATION",
|
"type": "YARN_APPLICATION",
|
||||||
"createdtime": 0,
|
"createdtime": 1460419580171,
|
||||||
"info": {
|
"info": {
|
||||||
"UID": "test-cluster!sjlee!ds-date!1460420587974!application_1460419579913_0002"
|
"UID": "test-cluster!sjlee!ds-date!1460419580171!application_1460419579913_0002"
|
||||||
},
|
},
|
||||||
"configs": {},
|
"configs": {},
|
||||||
"isrelatedto": {},
|
"isrelatedto": {},
|
||||||
@ -551,26 +852,289 @@ Response:
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
You can also provide per-framework entity type to query for them. For example,
|
### Response Codes
|
||||||
|
|
||||||
HTTP request:
|
1. If successful, a HTTP 200 (OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/DS_APP_ATTEMPT
|
|
||||||
|
|
||||||
Response:
|
## <a name="REST_API_LIST_APP"></a>Query app
|
||||||
|
|
||||||
|
With this API, you can query a single YARN application identified by cluster and application ID. If REST endpoint without cluster name
|
||||||
|
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
|
||||||
|
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
|
||||||
|
flow context information based on cluster and app id.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/apps/{app id}
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `userid` - If specified, only applications belonging to this user will be returned. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowname` - Only applications belonging to this flowname will be returned. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowrunid` - Only applications belonging to this flow run id will be returned. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `confsToRetrieve` - If specified, defines which configs to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
confsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for the
|
||||||
|
simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If confsToRetrieve is specified, configs will be retrieved irrespective of whether `CONFIGS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. Possible values for fields can be `EVENTS`, `INFO`, `CONFIGS`, `METRICS`, `RELATES_TO`,
|
||||||
|
`IS_RELATED_TO` and `ALL`. All fields will be retrieved if `ALL` is specified. Multiple fields can be specified as a comma-separated list.
|
||||||
|
If fields is not specified, in response, app id, type (equivalent to YARN_APPLICATION), app createdtime and UID in info field will be returned.
|
||||||
|
1. `metricsLimit` - If specified, defines the number of metrics to return. Considered only if fields contains METRICS/ALL
|
||||||
|
or metricsToRetrieve is specified. Ignored otherwise. The maximum possible value for metricsLimit can be maximum value of
|
||||||
|
Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricsLimit will be
|
||||||
|
considered as 1 i.e. latest single value of metric(s) will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
|
{
|
||||||
|
"metrics": [],
|
||||||
|
"events": [],
|
||||||
|
"id": "application_1460419579913_0002",
|
||||||
|
"type": "YARN_APPLICATION",
|
||||||
|
"createdtime": 1460419580171,
|
||||||
|
"info": {
|
||||||
|
"UID": "test-cluster!sjlee!ds-date!1460419580171!application_1460419579913_0002"
|
||||||
|
},
|
||||||
|
"configs": {},
|
||||||
|
"isrelatedto": {},
|
||||||
|
"relatesto": {}
|
||||||
|
}
|
||||||
|
|
||||||
|
### Response Codes
|
||||||
|
|
||||||
|
1. If successful, a HTTP 200(OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. If flow context information cannot be retrieved or application for the given app id cannot be found, HTTP 404 (Not Found) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
|
## <a name="REST_API_LIST_ENTITIES"></a>Query generic entities
|
||||||
|
|
||||||
|
With this API, you can query generic entities identified by cluster ID, application ID and per-framework entity type. If REST endpoint without cluster name
|
||||||
|
is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
|
||||||
|
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
|
||||||
|
flow context information based on cluster and app id. If number of matching entities are more than the limit, the most recent entities up to the limit
|
||||||
|
will be returned. This endpoint can be used to query containers, application attempts or any other generic entity which clients put into the backend.
|
||||||
|
For instance, we can query containers by specifying entity type as `YARN_CONTAINER` and application attempts by specifying entity type as `YARN_APPLICATION_ATTEMPT`.
|
||||||
|
If none of the entities match the predicates, an empty list will be returned.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type}
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/apps/{app id}/entities/{entity type}
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `userid` - If specified, only entities belonging to this user will be returned. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowname` - If specified, only entities belonging to this flowname will be returned. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowrunid` - If specified, only entities belonging to this flow run id will be returned. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `limit` - If specified, defines the number of entities to return. The maximum possible value for limit is maximum value of Long. If it is not specified
|
||||||
|
or has a value less than 0, then limit will be considered as 100.
|
||||||
|
1. `createdTimeStart` - If specified, then only entities created after this timestamp are returned.
|
||||||
|
1. `createdTimeEnd` - If specified, then only entities created before this timestamp are returned.
|
||||||
|
1. `relatesTo` - If specified, matched entities must relate to or not relate to given entities associated with a entity type.
|
||||||
|
relatesto is represented as an expression of the form :<br/>
|
||||||
|
"(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...) <op> !(<entitytype>:<entityid>:<entityid>...,<entitytype>:<entityid>:<entityid>...)".<br/>
|
||||||
|
If relatesTo expression has entity type - entity id(s) relations specified within enclosing brackets proceeding "!", this means entities with
|
||||||
|
these relations in its relatesTo field, will not be returned. For expressions or subexpressions without "!", all entities which have the specified
|
||||||
|
relations in its relatesTo field, will be returned. "op" is a logical operator and can be either AND or OR. entity type can be followed by any number
|
||||||
|
of entity id(s). And we can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : relatesTo can be "(((type1:id1:id2:id3,type3:id9) AND !(type2:id7:id8)) OR (type1:id4))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `isRelatedTo` - If specified, matched entities must be related to or not related to given entities associated with a entity type. isRelatedTo is
|
||||||
|
represented in the same form as relatesTo.
|
||||||
|
1. `infofilters` - If specified, matched entities must have exact matches to the given info key and must be either equal or not equal to
|
||||||
|
given value. The info key is a string but value can be any object. infofilters are represented as an expression of the form :<br/>
|
||||||
|
"(<key> <compareop> <value>) <op> (<key> <compareop> <value>)".<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne" or "ene".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of key is not required for a match and "ene" means not equals but existence of key is
|
||||||
|
required. We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : infofilters can be "(((infokey1 eq value1) AND (infokey2 ne value1)) OR (infokey1 ene value3))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `conffilters` - If specified, matched entities must have exact matches to the given config name and must be either equal or not equal
|
||||||
|
to the given config value. Both the config name and value must be strings. conffilters are represented in the same form as infofilters.
|
||||||
|
1. `metricfilters` - If specified, matched entities must have exact matches to the given metric and satisfy the specified relation with the
|
||||||
|
metric value. Metric id must be a string and and metric value must be an integral value. metricfilters are represented as an expression of the form :<br/>
|
||||||
|
"(<metricid> <compareop> <metricvalue>) <op> (<metricid> <compareop> <metricvalue>)"<br/>
|
||||||
|
Here op can be either of AND or OR. And compareop can be either of "eq", "ne", "ene", "gt", "ge", "lt" and "le".<br/>
|
||||||
|
"eq" means equals, "ne" means not equals and existence of metric is not required for a match, "ene" means not equals but existence of metric is
|
||||||
|
required, "gt" means greater than, "ge" means greater than or equals, "lt" means less than and "le" means less than or equals. We can combine
|
||||||
|
any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : metricfilters can be "(((metric1 eq 50) AND (metric2 gt 40)) OR (metric1 lt 20))".<br/>
|
||||||
|
This in essence is an expression equivalent to "(metric1 == 50 AND metric2 > 40) OR (metric1 < 20)"<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `eventfilters` - If specified, matched entities must contain or not contain the given events depending on the expression. eventfilters is
|
||||||
|
represented as an expression of the form :<br/>
|
||||||
|
"(<eventid>,<eventid>) <op> !(<eventid>,<eventid>,<eventid>)".<br/>
|
||||||
|
Here, "!" means none of the comma-separated list of events within the enclosed brackets proceeding "!" must exist for a match to occur.
|
||||||
|
If "!" is not specified, the specified events within the enclosed brackets must exist. op is a logical operator and can be either AND or OR.
|
||||||
|
We can combine any number of ANDs' and ORs' to create complex expressions. Brackets can be used to club expressions together.<br/>
|
||||||
|
_For example_ : eventfilters can be "(((event1,event2) AND !(event4)) OR (event3,event7,event5))".<br/>
|
||||||
|
Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional for
|
||||||
|
the simple expression. Alternatively, expressions can be of the form:<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `confsToRetrieve` - If specified, defines which configs to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
confsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for
|
||||||
|
the simple expression. Alternatively, expressions can be of the form:<br/>
|
||||||
|
!(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If confsToRetrieve is specified, configs will be retrieved irrespective of whether `CONFIGS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. Possible values for fields can be `EVENTS`, `INFO`, `CONFIGS`, `METRICS`, `RELATES_TO`,
|
||||||
|
`IS_RELATED_TO` and `ALL`. All fields will be retrieved if `ALL` is specified. Multiple fields can be specified as a comma-separated list.
|
||||||
|
If fields is not specified, in response, entity id, entity type, createdtime and UID in info field will be returned.
|
||||||
|
1. `metricsLimit` - If specified, defines the number of metrics to return. Considered only if fields contains METRICS/ALL
|
||||||
|
or metricsToRetrieve is specified. Ignored otherwise. The maximum possible value for metricsLimit can be maximum value of
|
||||||
|
Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricsLimit will be
|
||||||
|
considered as 1 i.e. latest single value of metric(s) will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"metrics": [ ],
|
"metrics": [ ],
|
||||||
"events": [ ],
|
"events": [ ],
|
||||||
"id": "appattempt_1460419579913_0002_000001",
|
"type": "YARN_APPLICATION_ATTEMPT",
|
||||||
"type": "DS_APP_ATTEMPT",
|
"id": "appattempt_1465246237936_0001_000001",
|
||||||
"createdtime": 0,
|
"createdtime": 1465246358873,
|
||||||
"info": {
|
|
||||||
"UID": "test-cluster!application_1460419579913_0002!DS_APP_ATTEMPT!appattempt_1460419579913_0002_000001"
|
|
||||||
},
|
|
||||||
"configs": {},
|
|
||||||
"isrelatedto": { },
|
"isrelatedto": { },
|
||||||
|
"configs": { },
|
||||||
|
"info": {
|
||||||
|
"UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000001"
|
||||||
|
},
|
||||||
|
"relatesto": { }
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"metrics": [ ],
|
||||||
|
"events": [ ],
|
||||||
|
"type": "YARN_APPLICATION_ATTEMPT",
|
||||||
|
"id": "appattempt_1465246237936_0001_000002",
|
||||||
|
"createdtime": 1465246359045,
|
||||||
|
"isrelatedto": { },
|
||||||
|
"configs": { },
|
||||||
|
"info": {
|
||||||
|
"UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000002"
|
||||||
|
},
|
||||||
"relatesto": { }
|
"relatesto": { }
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
|
### Response Codes
|
||||||
|
|
||||||
|
1. If successful, a HTTP 200(OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. If flow context information cannot be retrieved, HTTP 404 (Not Found) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
|
## <a name="REST_API_LIST_ENTITY"></a>Query generic entity
|
||||||
|
|
||||||
|
With this API, you can query a specific generic entity identified by cluster ID, application ID, per-framework entity type and entity ID. If REST endpoint without
|
||||||
|
cluster name is used, cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. Flow context information i.e.
|
||||||
|
user, flow name and run id are not mandatory but if specified in query param can preclude the need for an additional operation to fetch
|
||||||
|
flow context information based on cluster and app id. This endpoint can be used to query a single container, application attempt or any other generic entity which
|
||||||
|
clients put into the backend. For instance, we can query a specific YARN container by specifying entity type as `YARN_CONTAINER` and giving entity ID as container ID.
|
||||||
|
Similarly, application attempt can be queried by specifying entity type as `YARN_APPLICATION_ATTEMPT` and entity ID being the application attempt ID.
|
||||||
|
|
||||||
|
### HTTP request:
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type}/{entity id}
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
GET /ws/v2/timeline/apps/{app id}/entities/{entity type}/{entity id}
|
||||||
|
|
||||||
|
### Query Parameters Supported:
|
||||||
|
|
||||||
|
1. `userid` - If specified, entity must belong to this user. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowname` - If specified, entity must belong to this flow name. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `flowrunid` - If specified, entity must belong to this flow run id. This query param must be specified along with userid and flowname query params, otherwise it will be ignored.
|
||||||
|
If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query.
|
||||||
|
1. `metricsToRetrieve` - If specified, defines which metrics to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
metricsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics matching any of the prefixes will be retrieved. Brackets are optional
|
||||||
|
for the simple expression. Alternatively, expressions can be of the form:<br/>
|
||||||
|
!(<metricprefix>,<metricprefix>,<metricprefix>,<metricprefix>...)<br/>
|
||||||
|
This specifies a comma separated list of metric id prefixes. Only metrics not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If metricsToRetrieve is specified, metrics will be retrieved irrespective of whether `METRICS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `confsToRetrieve` - If specified, defines which configs to retrieve or which ones not to retrieve and send back in response.
|
||||||
|
confsToRetrieve can be an expression of the form :<br/>
|
||||||
|
(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs matching any of the prefixes will be retrieved. Brackets are optional for
|
||||||
|
the simple expression. Alternatively, expressions can be of the form :<br/>
|
||||||
|
!(<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>,<config\_name\_prefix>...)<br/>
|
||||||
|
This specifies a comma separated list of config name prefixes. Only configs not matching any of the prefixes will be retrieved.<br/>
|
||||||
|
If confsToRetrieve is specified, configs will be retrieved irrespective of whether `CONFIGS` is specified in fields query param
|
||||||
|
or not. Please note that URL unsafe characters such as spaces will have to be suitably encoded.
|
||||||
|
1. `fields` - Specifies which fields to retrieve. Possible values for fields can be `EVENTS`, `INFO`, `CONFIGS`, `METRICS`, `RELATES_TO`,
|
||||||
|
`IS_RELATED_TO` and `ALL`. All fields will be retrieved if `ALL` is specified. Multiple fields can be specified as a comma-separated list.
|
||||||
|
If fields is not specified, in response, entity id, entity type, createdtime and UID in info field will be returned.
|
||||||
|
1. `metricsLimit` - If specified, defines the number of metrics to return. Considered only if fields contains METRICS/ALL
|
||||||
|
or metricsToRetrieve is specified. Ignored otherwise. The maximum possible value for metricsLimit can be maximum value of
|
||||||
|
Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricsLimit will be
|
||||||
|
considered as 1 i.e. latest single value of metric(s) will be returned.
|
||||||
|
|
||||||
|
### Example JSON Response:
|
||||||
|
|
||||||
|
{
|
||||||
|
"metrics": [ ],
|
||||||
|
"events": [ ],
|
||||||
|
"type": "YARN_APPLICATION_ATTEMPT",
|
||||||
|
"id": "appattempt_1465246237936_0001_000001",
|
||||||
|
"createdtime": 1465246358873,
|
||||||
|
"isrelatedto": { },
|
||||||
|
"configs": { },
|
||||||
|
"info": {
|
||||||
|
"UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000001"
|
||||||
|
},
|
||||||
|
"relatesto": { }
|
||||||
|
}
|
||||||
|
|
||||||
|
### Response Codes
|
||||||
|
|
||||||
|
1. If successful, a HTTP 200 (OK) response is returned.
|
||||||
|
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned.
|
||||||
|
1. If flow context information cannot be retrieved or entity for the given entity id cannot be found, HTTP 404 (Not Found) is returned.
|
||||||
|
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user