102 KiB
102 KiB
ResourceManager REST API's.
- Overview
- Cluster Information API
- Cluster Metrics API
- Cluster Scheduler API
- Cluster Applications API
- Cluster Application Statistics API
- Cluster Application API
- Cluster Application Attempts API
- Cluster Nodes API
- Cluster Node API
- Cluster Writeable APIs
- Cluster New Application API
- Cluster Applications API(Submit Application)
- Cluster Application State API
- Cluster Application Queue API
- Cluster Delegation Tokens API
Overview
The ResourceManager REST API's allow the user to get information about the cluster - status on the cluster, metrics on the cluster, scheduler information, information about nodes in the cluster, and information about applications on the cluster.
Cluster Information API
The cluster information resource provides overall information about the cluster.
URI
Both of the following URI's give you the cluster information.
* http://<rm http address:port>/ws/v1/cluster
* http://<rm http address:port>/ws/v1/cluster/info
HTTP Operations Supported
* GET
Query Parameters Supported
None
Elements of the clusterInfo object
| Item | Data Type | Description |
|---|---|---|
| id | long | The cluster id |
| startedOn | long | The time the cluster started (in ms since epoch) |
| state | string | The ResourceManager state - valid values are: NOTINITED, INITED, STARTED, STOPPED |
| haState | string | The ResourceManager HA state - valid values are: INITIALIZING, ACTIVE, STANDBY, STOPPED |
| resourceManagerVersion | string | Version of the ResourceManager |
| resourceManagerBuildVersion | string | ResourceManager build string with build version, user, and checksum |
| resourceManagerVersionBuiltOn | string | Timestamp when ResourceManager was built (in ms since epoch) |
| hadoopVersion | string | Version of hadoop common |
| hadoopBuildVersion | string | Hadoop common build string with build version, user, and checksum |
| hadoopVersionBuiltOn | string | Timestamp when hadoop common was built(in ms since epoch) |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/info
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"clusterInfo":
{
"id":1324053971963,
"startedOn":1324053971963,
"state":"STARTED",
"resourceManagerVersion":"0.23.1-SNAPSHOT",
"resourceManagerBuildVersion":"0.23.1-SNAPSHOT from 1214049 by user1 source checksum 050cd664439d931c8743a6428fd6a693",
"resourceManagerVersionBuiltOn":"Tue Dec 13 22:12:48 CST 2011",
"hadoopVersion":"0.23.1-SNAPSHOT",
"hadoopBuildVersion":"0.23.1-SNAPSHOT from 1214049 by user1 source checksum 11458df3bb77342dca5f917198fad328",
"hadoopVersionBuiltOn":"Tue Dec 13 22:12:26 CST 2011"
}
}
XML response
HTTP Request:
Accept: application/xml
GET http://<rm http address:port>/ws/v1/cluster/info
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 712
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<clusterInfo>
<id>1324053971963</id>
<startedOn>1324053971963</startedOn>
<state>STARTED</state>
<resourceManagerVersion>0.23.1-SNAPSHOT</resourceManagerVersion>
<resourceManagerBuildVersion>0.23.1-SNAPSHOT from 1214049 by user1 source checksum 050cd664439d931c8743a6428fd6a693</resourceManagerBuildVersion>
<resourceManagerVersionBuiltOn>Tue Dec 13 22:12:48 CST 2011</resourceManagerVersionBuiltOn>
<hadoopVersion>0.23.1-SNAPSHOT</hadoopVersion>
<hadoopBuildVersion>0.23.1-SNAPSHOT from 1214049 by user1 source checksum 11458df3bb77342dca5f917198fad328</hadoopBuildVersion>
<hadoopVersionBuiltOn>Tue Dec 13 22:12:48 CST 2011</hadoopVersionBuiltOn>
</clusterInfo>
Cluster Metrics API
The cluster metrics resource provides some overall metrics about the cluster. More detailed metrics should be retrieved from the jmx interface.
URI
* http://<rm http address:port>/ws/v1/cluster/metrics
HTTP Operations Supported
* GET
Query Parameters Supported
None
Elements of the clusterMetrics object
| Item | Data Type | Description |
|---|---|---|
| appsSubmitted | int | The number of applications submitted |
| appsCompleted | int | The number of applications completed |
| appsPending | int | The number of applications pending |
| appsRunning | int | The number of applications running |
| appsFailed | int | The number of applications failed |
| appsKilled | int | The number of applications killed |
| reservedMB | long | The amount of memory reserved in MB |
| availableMB | long | The amount of memory available in MB |
| allocatedMB | long | The amount of memory allocated in MB |
| totalMB | long | The amount of total memory in MB |
| reservedVirtualCores | long | The number of reserved virtual cores |
| availableVirtualCores | long | The number of available virtual cores |
| allocatedVirtualCores | long | The number of allocated virtual cores |
| totalVirtualCores | long | The total number of virtual cores |
| containersAllocated | int | The number of containers allocated |
| containersReserved | int | The number of containers reserved |
| containersPending | int | The number of containers pending |
| totalNodes | int | The total number of nodes |
| activeNodes | int | The number of active nodes |
| lostNodes | int | The number of lost nodes |
| unhealthyNodes | int | The number of unhealthy nodes |
| decommissionedNodes | int | The number of nodes decommissioned |
| rebootedNodes | int | The number of nodes rebooted |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/metrics
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"clusterMetrics":
{
"appsSubmitted":0,
"appsCompleted":0,
"appsPending":0,
"appsRunning":0,
"appsFailed":0,
"appsKilled":0,
"reservedMB":0,
"availableMB":17408,
"allocatedMB":0,
"reservedVirtualCores":0,
"availableVirtualCores":7,
"allocatedVirtualCores":1,
"containersAllocated":0,
"containersReserved":0,
"containersPending":0,
"totalMB":17408,
"totalVirtualCores":8,
"totalNodes":1,
"lostNodes":0,
"unhealthyNodes":0,
"decommissionedNodes":0,
"rebootedNodes":0,
"activeNodes":1
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/metrics
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 432
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<clusterMetrics>
<appsSubmitted>0</appsSubmitted>
<appsCompleted>0</appsCompleted>
<appsPending>0</appsPending>
<appsRunning>0</appsRunning>
<appsFailed>0</appsFailed>
<appsKilled>0</appsKilled>
<reservedMB>0</reservedMB>
<availableMB>17408</availableMB>
<allocatedMB>0</allocatedMB>
<reservedVirtualCores>0</reservedVirtualCores>
<availableVirtualCores>7</availableVirtualCores>
<allocatedVirtualCores>1</allocatedVirtualCores>
<containersAllocated>0</containersAllocated>
<containersReserved>0</containersReserved>
<containersPending>0</containersPending>
<totalMB>17408</totalMB>
<totalVirtualCores>8</totalVirtualCores>
<totalNodes>1</totalNodes>
<lostNodes>0</lostNodes>
<unhealthyNodes>0</unhealthyNodes>
<decommissionedNodes>0</decommissionedNodes>
<rebootedNodes>0</rebootedNodes>
<activeNodes>1</activeNodes>
</clusterMetrics>
Cluster Scheduler API
A scheduler resource contains information about the current scheduler configured in a cluster. It currently supports the Fifo, Capacity and Fair Scheduler. You will get different information depending on which scheduler is configured so be sure to look at the type information.
URI
* http://<rm http address:port>/ws/v1/cluster/scheduler
HTTP Operations Supported
* GET
Query Parameters Supported
None
Capacity Scheduler API
The capacity scheduler supports hierarchical queues. This one request will print information about all the queues and any subqueues they have. Queues that can actually have jobs submitted to them are referred to as leaf queues. These queues have additional data associated with them.
Elements of the schedulerInfo object
| Item | Data Type | Description |
|---|---|---|
| type | string | Scheduler type - capacityScheduler |
| capacity | float | Configured queue capacity in percentage relative to its parent queue |
| usedCapacity | float | Used queue capacity in percentage |
| maxCapacity | float | Configured maximum queue capacity in percentage relative to its parent queue |
| queueName | string | Name of the queue |
| queues | array of queues(JSON)/zero or more queue objects(XML) | A collection of queue resources |
Elements of the queues object for a Parent queue
| Item | Data Type | Description |
|---|---|---|
| capacity | float | Configured queue capacity in percentage relative to its parent queue |
| usedCapacity | float | Used queue capacity in percentage |
| maxCapacity | float | Configured maximum queue capacity in percentage relative to its parent queue |
| absoluteCapacity | float | Absolute capacity percentage this queue can use of entire cluster |
| absoluteMaxCapacity | float | Absolute maximum capacity percentage this queue can use of the entire cluster |
| absoluteUsedCapacity | float | Absolute used capacity percentage this queue is using of the entire cluster |
| numApplications | int | The number of applications currently in the queue |
| usedResources | string | A string describing the current resources used by the queue |
| queueName | string | The name of the queue |
| state | string of QueueState | The state of the queue |
| queues | array of queues(JSON)/zero or more queue objects(XML) | A collection of sub-queue information. Omitted if the queue has no sub-queues. |
| resourcesUsed | A single resource object | The total amount of resources used by this queue |
Elements of the queues object for a Leaf queue - contains all the elements in parent except 'queues' plus the following:
| Item | Data Type | Description |
|---|---|---|
| type | String | type of the queue - capacitySchedulerLeafQueueInfo |
| numActiveApplications | int | The number of active applications in this queue |
| numPendingApplications | int | The number of pending applications in this queue |
| numContainers | int | The number of containers being used |
| maxApplications | int | The maximum number of applications this queue can have |
| maxApplicationsPerUser | int | The maximum number of applications per user this queue can have |
| maxActiveApplications | int | The maximum number of active applications this queue can have |
| maxActiveApplicationsPerUser | int | The maximum number of active applications per user this queue can have |
| userLimit | int | The minimum user limit percent set in the configuration |
| userLimitFactor | float | The user limit factor set in the configuration |
| users | array of users(JSON)/zero or more user objects(XML) | A collection of user objects containing resources used |
Elements of the user object for users:
| Item | Data Type | Description |
|---|---|---|
| username | String | The username of the user using the resources |
| resourcesUsed | A single resource object | The amount of resources used by the user in this queue |
| numActiveApplications | int | The number of active applications for this user in this queue |
| numPendingApplications | int | The number of pending applications for this user in this queue |
Elements of the resource object for resourcesUsed in user and queues:
| Item | Data Type | Description |
|---|---|---|
| memory | int | The amount of memory used (in MB) |
| vCores | int | The number of virtual cores |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"scheduler": {
"schedulerInfo": {
"capacity": 100.0,
"maxCapacity": 100.0,
"queueName": "root",
"queues": {
"queue": [
{
"absoluteCapacity": 10.5,
"absoluteMaxCapacity": 50.0,
"absoluteUsedCapacity": 0.0,
"capacity": 10.5,
"maxCapacity": 50.0,
"numApplications": 0,
"queueName": "a",
"queues": {
"queue": [
{
"absoluteCapacity": 3.15,
"absoluteMaxCapacity": 25.0,
"absoluteUsedCapacity": 0.0,
"capacity": 30.000002,
"maxCapacity": 50.0,
"numApplications": 0,
"queueName": "a1",
"queues": {
"queue": [
{
"absoluteCapacity": 2.6775,
"absoluteMaxCapacity": 25.0,
"absoluteUsedCapacity": 0.0,
"capacity": 85.0,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 1,
"maxApplications": 267,
"maxApplicationsPerUser": 267,
"maxCapacity": 100.0,
"numActiveApplications": 0,
"numApplications": 0,
"numContainers": 0,
"numPendingApplications": 0,
"queueName": "a1a",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 1.0,
"users": null
},
{
"absoluteCapacity": 0.47250003,
"absoluteMaxCapacity": 25.0,
"absoluteUsedCapacity": 0.0,
"capacity": 15.000001,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 1,
"maxApplications": 47,
"maxApplicationsPerUser": 47,
"maxCapacity": 100.0,
"numActiveApplications": 0,
"numApplications": 0,
"numContainers": 0,
"numPendingApplications": 0,
"queueName": "a1b",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 1.0,
"users": null
}
]
},
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>"
},
{
"absoluteCapacity": 7.35,
"absoluteMaxCapacity": 50.0,
"absoluteUsedCapacity": 0.0,
"capacity": 70.0,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 100,
"maxApplications": 735,
"maxApplicationsPerUser": 73500,
"maxCapacity": 100.0,
"numActiveApplications": 0,
"numApplications": 0,
"numContainers": 0,
"numPendingApplications": 0,
"queueName": "a2",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 100.0,
"users": null
}
]
},
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>"
},
{
"absoluteCapacity": 89.5,
"absoluteMaxCapacity": 100.0,
"absoluteUsedCapacity": 0.0,
"capacity": 89.5,
"maxCapacity": 100.0,
"numApplications": 2,
"queueName": "b",
"queues": {
"queue": [
{
"absoluteCapacity": 53.7,
"absoluteMaxCapacity": 100.0,
"absoluteUsedCapacity": 0.0,
"capacity": 60.000004,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 100,
"maxApplications": 5370,
"maxApplicationsPerUser": 537000,
"maxCapacity": 100.0,
"numActiveApplications": 1,
"numApplications": 2,
"numContainers": 0,
"numPendingApplications": 1,
"queueName": "b1",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 100.0,
"users": {
"user": [
{
"numActiveApplications": 0,
"numPendingApplications": 1,
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"username": "user2"
},
{
"numActiveApplications": 1,
"numPendingApplications": 0,
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"username": "user1"
}
]
}
},
{
"absoluteCapacity": 35.3525,
"absoluteMaxCapacity": 100.0,
"absoluteUsedCapacity": 0.0,
"capacity": 39.5,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 100,
"maxApplications": 3535,
"maxApplicationsPerUser": 353500,
"maxCapacity": 100.0,
"numActiveApplications": 0,
"numApplications": 0,
"numContainers": 0,
"numPendingApplications": 0,
"queueName": "b2",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 100.0,
"users": null
},
{
"absoluteCapacity": 0.4475,
"absoluteMaxCapacity": 100.0,
"absoluteUsedCapacity": 0.0,
"capacity": 0.5,
"maxActiveApplications": 1,
"maxActiveApplicationsPerUser": 100,
"maxApplications": 44,
"maxApplicationsPerUser": 4400,
"maxCapacity": 100.0,
"numActiveApplications": 0,
"numApplications": 0,
"numContainers": 0,
"numPendingApplications": 0,
"queueName": "b3",
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"type": "capacitySchedulerLeafQueueInfo",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>",
"userLimit": 100,
"userLimitFactor": 100.0,
"users": null
}
]
},
"resourcesUsed": {
"memory": 0,
"vCores": 0
},
"state": "RUNNING",
"usedCapacity": 0.0,
"usedResources": "<memory:0, vCores:0>"
}
]
},
"type": "capacityScheduler",
"usedCapacity": 0.0
}
}
}
```json
**XML response**
HTTP Request:
Accept: application/xml
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 5778
Server: Jetty(6.1.26)
Response Body:
```xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<scheduler>
<schedulerInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="capacityScheduler">
<capacity>100.0</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<queueName>root</queueName>
<queues>
<queue>
<capacity>10.5</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>50.0</maxCapacity>
<absoluteCapacity>10.5</absoluteCapacity>
<absoluteMaxCapacity>50.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>a</queueName>
<state>RUNNING</state>
<queues>
<queue>
<capacity>30.000002</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>50.0</maxCapacity>
<absoluteCapacity>3.15</absoluteCapacity>
<absoluteMaxCapacity>25.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>a1</queueName>
<state>RUNNING</state>
<queues>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>85.0</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>2.6775</absoluteCapacity>
<absoluteMaxCapacity>25.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>a1a</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>267</maxApplications>
<maxApplicationsPerUser>267</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>1</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>1.0</userLimitFactor>
</queue>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>15.000001</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>0.47250003</absoluteCapacity>
<absoluteMaxCapacity>25.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>a1b</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>47</maxApplications>
<maxApplicationsPerUser>47</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>1</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>1.0</userLimitFactor>
</queue>
</queues>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
</queue>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>70.0</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>7.35</absoluteCapacity>
<absoluteMaxCapacity>50.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>a2</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>735</maxApplications>
<maxApplicationsPerUser>73500</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>100</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>100.0</userLimitFactor>
</queue>
</queues>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
</queue>
<queue>
<capacity>89.5</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>89.5</absoluteCapacity>
<absoluteMaxCapacity>100.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>2</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>b</queueName>
<state>RUNNING</state>
<queues>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>60.000004</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>53.7</absoluteCapacity>
<absoluteMaxCapacity>100.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>2</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>b1</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>1</numActiveApplications>
<numPendingApplications>1</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>5370</maxApplications>
<maxApplicationsPerUser>537000</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>100</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users>
<user>
<username>user2</username>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numPendingApplications>1</numPendingApplications>
<numActiveApplications>0</numActiveApplications>
</user>
<user>
<username>user1</username>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numPendingApplications>0</numPendingApplications>
<numActiveApplications>1</numActiveApplications>
</user>
</users>
<userLimitFactor>100.0</userLimitFactor>
</queue>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>39.5</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>35.3525</absoluteCapacity>
<absoluteMaxCapacity>100.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>b2</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>3535</maxApplications>
<maxApplicationsPerUser>353500</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>100</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>100.0</userLimitFactor>
</queue>
<queue xsi:type="capacitySchedulerLeafQueueInfo">
<capacity>0.5</capacity>
<usedCapacity>0.0</usedCapacity>
<maxCapacity>100.0</maxCapacity>
<absoluteCapacity>0.4475</absoluteCapacity>
<absoluteMaxCapacity>100.0</absoluteMaxCapacity>
<absoluteUsedCapacity>0.0</absoluteUsedCapacity>
<numApplications>0</numApplications>
<usedResources><memory:0, vCores:0></usedResources>
<queueName>b3</queueName>
<state>RUNNING</state>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
<numActiveApplications>0</numActiveApplications>
<numPendingApplications>0</numPendingApplications>
<numContainers>0</numContainers>
<maxApplications>44</maxApplications>
<maxApplicationsPerUser>4400</maxApplicationsPerUser>
<maxActiveApplications>1</maxActiveApplications>
<maxActiveApplicationsPerUser>100</maxActiveApplicationsPerUser>
<userLimit>100</userLimit>
<users/>
<userLimitFactor>100.0</userLimitFactor>
</queue>
</queues>
<resourcesUsed>
<memory>0</memory>
<vCores>0</vCores>
</resourcesUsed>
</queue>
</queues>
</schedulerInfo>
</scheduler>
Fifo Scheduler API
Elements of the schedulerInfo object
| Item | Data Type | Description |
|---|---|---|
| type | string | Scheduler type - fifoScheduler |
| capacity | float | Queue capacity in percentage |
| usedCapacity | float | Used queue capacity in percentage |
| qstate | string | State of the queue - valid values are: STOPPED, RUNNING |
| minQueueMemoryCapacity | int | Minimum queue memory capacity |
| maxQueueMemoryCapacity | int | Maximum queue memory capacity |
| numNodes | int | The total number of nodes |
| usedNodeCapacity | int | The used node capacity |
| availNodeCapacity | int | The available node capacity |
| totalNodeCapacity | int | The total node capacity |
| numContainers | int | The number of containers |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"scheduler":
{
"schedulerInfo":
{
"type":"fifoScheduler",
"capacity":1,
"usedCapacity":"NaN",
"qstate":"RUNNING",
"minQueueMemoryCapacity":1024,
"maxQueueMemoryCapacity":10240,
"numNodes":0,
"usedNodeCapacity":0,
"availNodeCapacity":0,
"totalNodeCapacity":0,
"numContainers":0
}
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 432
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<scheduler>
<schedulerInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="fifoScheduler">
<capacity>1.0</capacity>
<usedCapacity>NaN</usedCapacity>
<qstate>RUNNING</qstate>
<minQueueMemoryCapacity>1024</minQueueMemoryCapacity>
<maxQueueMemoryCapacity>10240</maxQueueMemoryCapacity>
<numNodes>0</numNodes>
<usedNodeCapacity>0</usedNodeCapacity>
<availNodeCapacity>0</availNodeCapacity>
<totalNodeCapacity>0</totalNodeCapacity>
<numContainers>0</numContainers>
</schedulerInfo>
</scheduler>
Fair Scheduler API
Elements of the schedulerInfo object
| Item | Data Type | Description |
|---|---|---|
| type | string | Scheduler type - fairScheduler |
| rootQueue | The root queue object | A collection of root queue resources |
Elements of the root queue object
| Item | Data Type | Description |
|---|---|---|
| maxApps | int | The maximum number of applications the queue can have |
| minResources | A single resource object | The configured minimum resources that are guaranteed to the queue |
| maxResources | A single resource object | The configured maximum resources that are allowed to the queue |
| usedResources | A single resource object | The sum of resources allocated to containers within the queue |
| fairResources | A single resource object | The queue's fair share of resources |
| clusterResources | A single resource object | The capacity of the cluster |
| queueName | string | The name of the queue |
| schedulingPolicy | string | The name of the scheduling policy used by the queue |
| childQueues | array of queues(JSON)/queue objects(XML) | A collection of sub-queue information. Omitted if the queue has no childQueues. |
Elements of the queues object for a Leaf queue - contains all the elements in parent except 'childQueues' plus the following
| Item | Data Type | Description |
|---|---|---|
| type | string | type of the queue - fairSchedulerLeafQueueInfo |
| numActiveApps | int | The number of active applications in this queue |
| numPendingApps | int | The number of pending applications in this queue |
Elements of the resource object for resourcesUsed in queues
| Item | Data Type | Description |
|---|---|---|
| memory | int | The amount of memory used (in MB) |
| vCores | int | The number of virtual cores |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"scheduler": {
"schedulerInfo": {
"rootQueue": {
"childQueues": {
"queue": [
{
"clusterResources": {
"memory": 8192,
"vCores": 8
},
"fairResources": {
"memory": 0,
"vCores": 0
},
"maxApps": 2147483647,
"maxResources": {
"memory": 8192,
"vCores": 8
},
"minResources": {
"memory": 0,
"vCores": 0
},
"numActiveApps": 0,
"numPendingApps": 0,
"queueName": "root.default",
"schedulingPolicy": "fair",
"type": "fairSchedulerLeafQueueInfo",
"usedResources": {
"memory": 0,
"vCores": 0
}
},
{
"childQueues": {
"queue": [
{
"clusterResources": {
"memory": 8192,
"vCores": 8
},
"fairResources": {
"memory": 10000,
"vCores": 0
},
"maxApps": 2147483647,
"maxResources": {
"memory": 8192,
"vCores": 8
},
"minResources": {
"memory": 5000,
"vCores": 0
},
"numActiveApps": 0,
"numPendingApps": 0,
"queueName": "root.sample_queue.sample_sub_queue",
"schedulingPolicy": "fair",
"type": "fairSchedulerLeafQueueInfo",
"usedResources": {
"memory": 0,
"vCores": 0
}
}
]
},
"clusterResources": {
"memory": 8192,
"vCores": 8
},
"fairResources": {
"memory": 10000,
"vCores": 0
},
"maxApps": 50,
"maxResources": {
"memory": 8192,
"vCores": 0
},
"minResources": {
"memory": 10000,
"vCores": 0
},
"queueName": "root.sample_queue",
"schedulingPolicy": "fair",
"usedResources": {
"memory": 0,
"vCores": 0
}
}
],
},
"clusterResources": {
"memory": 8192,
"vCores": 8
},
"fairResources": {
"memory": 8192,
"vCores": 8
},
"maxApps": 2147483647,
"maxResources": {
"memory": 8192,
"vCores": 8
},
"minResources": {
"memory": 0,
"vCores": 0
},
"queueName": "root",
"schedulingPolicy": "fair",
"usedResources": {
"memory": 0,
"vCores": 0
}
},
"type": "fairScheduler"
}
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/scheduler
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2321
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<scheduler>
<schedulerInfo xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="fairScheduler">
<rootQueue>
<maxApps>2147483647</maxApps>
<minResources>
<memory>0</memory>
<vCores>0</vCores>
</minResources>
<maxResources>
<memory>8192</memory>
<vCores>8</vCores>
</maxResources>
<usedResources>
<memory>0</memory>
<vCores>0</vCores>
</usedResources>
<fairResources>
<memory>8192</memory>
<vCores>8</vCores>
</fairResources>
<clusterResources>
<memory>8192</memory>
<vCores>8</vCores>
</clusterResources>
<queueName>root</queueName>
<schedulingPolicy>fair</schedulingPolicy>
<childQueues>
<queue xsi:type="fairSchedulerLeafQueueInfo">
<maxApps>2147483647</maxApps>
<minResources>
<memory>0</memory>
<vCores>0</vCores>
</minResources>
<maxResources>
<memory>8192</memory>
<vCores>8</vCores>
</maxResources>
<usedResources>
<memory>0</memory>
<vCores>0</vCores>
</usedResources>
<fairResources>
<memory>0</memory>
<vCores>0</vCores>
</fairResources>
<clusterResources>
<memory>8192</memory>
<vCores>8</vCores>
</clusterResources>
<queueName>root.default</queueName>
<schedulingPolicy>fair</schedulingPolicy>
<numPendingApps>0</numPendingApps>
<numActiveApps>0</numActiveApps>
</queue>
<queue>
<maxApps>50</maxApps>
<minResources>
<memory>10000</memory>
<vCores>0</vCores>
</minResources>
<maxResources>
<memory>8192</memory>
<vCores>0</vCores>
</maxResources>
<usedResources>
<memory>0</memory>
<vCores>0</vCores>
</usedResources>
<fairResources>
<memory>10000</memory>
<vCores>0</vCores>
</fairResources>
<clusterResources>
<memory>8192</memory>
<vCores>8</vCores>
</clusterResources>
<queueName>root.sample_queue</queueName>
<schedulingPolicy>fair</schedulingPolicy>
<childQueues>
<queue xsi:type="fairSchedulerLeafQueueInfo">
<maxApps>2147483647</maxApps>
<minResources>
<memory>5000</memory>
<vCores>0</vCores>
</minResources>
<maxResources>
<memory>8192</memory>
<vCores>8</vCores>
</maxResources>
<usedResources>
<memory>0</memory>
<vCores>0</vCores>
</usedResources>
<fairResources>
<memory>10000</memory>
<vCores>0</vCores>
</fairResources>
<clusterResources>
<memory>8192</memory>
<vCores>8</vCores>
</clusterResources>
<queueName>root.sample_queue.sample_sub_queue</queueName>
<schedulingPolicy>fair</schedulingPolicy>
<numPendingApps>0</numPendingApps>
<numActiveApps>0</numActiveApps>
</queue>
</childQueues>
</queue>
</childQueues>
</rootQueue>
</schedulerInfo>
</scheduler>
Cluster Applications API
With the Applications API, you can obtain a collection of resources, each of which represents an application. When you run a GET operation on this resource, you obtain a collection of Application Objects.
URI
* http://<rm http address:port>/ws/v1/cluster/apps
HTTP Operations Supported
* GET
Query Parameters Supported
Multiple parameters can be specified for GET operations. The started and finished times have a begin and end parameter to allow you to specify ranges. For example, one could request all applications that started between 1:00am and 2:00pm on 12/19/2011 with startedTimeBegin=1324256400&startedTimeEnd=1324303200. If the Begin parameter is not specified, it defaults to 0, and if the End parameter is not specified, it defaults to infinity.
* state [deprecated] - state of the application
* states - applications matching the given application states, specified as a comma-separated list.
* finalStatus - the final status of the application - reported by the application itself
* user - user name
* queue - queue name
* limit - total number of app objects to be returned
* startedTimeBegin - applications with start time beginning with this time, specified in ms since epoch
* startedTimeEnd - applications with start time ending with this time, specified in ms since epoch
* finishedTimeBegin - applications with finish time beginning with this time, specified in ms since epoch
* finishedTimeEnd - applications with finish time ending with this time, specified in ms since epoch
* applicationTypes - applications matching the given application types, specified as a comma-separated list.
* applicationTags - applications matching any of the given application tags, specified as a comma-separated list.
Elements of the apps (Applications) object
When you make a request for the list of applications, the information will be returned as a collection of app objects. See also Application API for syntax of the app object.
| Item | Data Type | Description |
|---|---|---|
| app | array of app objects(JSON)/zero or more application objects(XML) | The collection of application objects |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"apps":
{
"app":
[
{
"finishedTime" : 1326815598530,
"amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326815542473_0001_01_000001",
"trackingUI" : "History",
"state" : "FINISHED",
"user" : "user1",
"id" : "application_1326815542473_0001",
"clusterId" : 1326815542473,
"finalStatus" : "SUCCEEDED",
"amHostHttpAddress" : "host.domain.com:8042",
"progress" : 100,
"name" : "word count",
"startedTime" : 1326815573334,
"elapsedTime" : 25196,
"diagnostics" : "",
"trackingUrl" : "http://host.domain.com:8088/proxy/application_1326815542473_0001/jobhistory/job/job_1326815542473_1_1",
"queue" : "default",
"allocatedMB" : 0,
"allocatedVCores" : 0,
"runningContainers" : 0,
"applicationType" : "MAPREDUCE",
"applicationTags" : ""
"memorySeconds" : 151730,
"vcoreSeconds" : 103
},
{
"finishedTime" : 1326815789546,
"amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326815542473_0002_01_000001",
"trackingUI" : "History",
"state" : "FINISHED",
"user" : "user1",
"id" : "application_1326815542473_0002",
"clusterId" : 1326815542473,
"finalStatus" : "SUCCEEDED",
"amHostHttpAddress" : "host.domain.com:8042",
"progress" : 100,
"name" : "Sleep job",
"startedTime" : 1326815641380,
"elapsedTime" : 148166,
"diagnostics" : "",
"trackingUrl" : "http://host.domain.com:8088/proxy/application_1326815542473_0002/jobhistory/job/job_1326815542473_2_2",
"queue" : "default",
"allocatedMB" : 0,
"allocatedVCores" : 0,
"runningContainers" : 1,
"applicationType" : "YARN",
"applicationTags" : "tag1"
"memorySeconds" : 640064,
"vcoreSeconds" : 442
}
]
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2459
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<apps>
<app>
<id>application_1326815542473_0001</id>
<user>user1</user>
<name>word count</name>
<applicationType>MAPREDUCE</applicationType>
<applicationTags></applicationTags>
<queue>default</queue>
<state>FINISHED</state>
<finalStatus>SUCCEEDED</finalStatus>
<progress>100.0</progress>
<trackingUI>History</trackingUI>
<trackingUrl>http://host.domain.com:8088/proxy/application_1326815542473_0001/jobhistory/job/job_1326815542473_1_1</trackingUrl>
<diagnostics/>
<clusterId>1326815542473</clusterId>
<startedTime>1326815573334</startedTime>
<finishedTime>1326815598530</finishedTime>
<elapsedTime>25196</elapsedTime>
<amContainerLogs>http://host.domain.com:8042/node/containerlogs/container_1326815542473_0001_01_000001</amContainerLogs>
<amHostHttpAddress>host.domain.com:8042</amHostHttpAddress>
<allocatedMB>0</allocatedMB>
<allocatedVCores>0</allocatedVCores>
<runningContainers>0</runningContainers>
<memorySeconds>151730</memorySeconds>
<vcoreSeconds>103</vcoreSeconds>
</app>
<app>
<id>application_1326815542473_0002</id>
<user>user1</user>
<name>Sleep job</name>
<applicationType>YARN</applicationType>
<applicationTags>tag1</applicationTags>
<queue>default</queue>
<state>FINISHED</state>
<finalStatus>SUCCEEDED</finalStatus>
<progress>100.0</progress>
<trackingUI>History</trackingUI>
<trackingUrl>http://host.domain.com:8088/proxy/application_1326815542473_0002/jobhistory/job/job_1326815542473_2_2</trackingUrl>
<diagnostics/>
<clusterId>1326815542473</clusterId>
<startedTime>1326815641380</startedTime>
<finishedTime>1326815789546</finishedTime>
<elapsedTime>148166</elapsedTime>
<amContainerLogs>http://host.domain.com:8042/node/containerlogs/container_1326815542473_0002_01_000001</amContainerLogs>
<amHostHttpAddress>host.domain.com:8042</amHostHttpAddress>
<allocatedMB>0</allocatedMB>
<allocatedVCores>0</allocatedVCores>
<runningContainers>0</runningContainers>
<memorySeconds>640064</memorySeconds>
<vcoreSeconds>442</vcoreSeconds>
</app>
</apps>
Cluster Application Statistics API
With the Application Statistics API, you can obtain a collection of triples, each of which contains the application type, the application state and the number of applications of this type and this state in ResourceManager context. Note that with the performance concern, we currently only support at most one applicationType per query. We may support multiple applicationTypes per query as well as more statistics in the future. When you run a GET operation on this resource, you obtain a collection of statItem objects.
URI
* http://<rm http address:port>/ws/v1/cluster/appstatistics
HTTP Operations Supported
* GET
Query Parameters Required
Two paramters can be specified. The parameters are case insensitive.
* states - states of the applications, specified as a comma-separated list. If states is not provided, the API will enumerate all application states and return the counts of them.
* applicationTypes - types of the applications, specified as a comma-separated list. If applicationTypes is not provided, the API will count the applications of any application type. In this case, the response shows * to indicate any application type. Note that we only support at most one applicationType temporarily. Otherwise, users will expect an BadRequestException.
Elements of the appStatInfo (statItems) object
When you make a request for the list of statistics items, the information will be returned as a collection of statItem objects
| Item | Data Type | Description |
|---|---|---|
| statItem | array of statItem objects(JSON)/zero or more statItem objects(XML) | The collection of statItem objects |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/appstatistics?states=accepted,running,finished&applicationTypes=mapreduce
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"appStatInfo":
{
"statItem":
[
{
"state" : "accepted",
"type" : "mapreduce",
"count" : 4
},
{
"state" : "running",
"type" : "mapreduce",
"count" : 1
},
{
"state" : "finished",
"type" : "mapreduce",
"count" : 7
}
]
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/appstatistics?states=accepted,running,finished&applicationTypes=mapreduce
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2459
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appStatInfo>
<statItem>
<state>accepted</state>
<type>mapreduce</type>
<count>4</count>
</statItem>
<statItem>
<state>running</state>
<type>mapreduce</type>
<count>1</count>
</statItem>
<statItem>
<state>finished</state>
<type>mapreduce</type>
<count>7</count>
</statItem>
</appStatInfo>
Cluster Application API
An application resource contains information about a particular application that was submitted to a cluster.
URI
Use the following URI to obtain an app object, from a application identified by the appid value.
* http://<rm http address:port>/ws/v1/cluster/apps/{appid}
HTTP Operations Supported
* GET
Query Parameters Supported
None
Elements of the app (Application) object
Note that depending on security settings a user might not be able to see all the fields.
| Item | Data Type | Description |
|---|---|---|
| id | string | The application id |
| user | string | The user who started the application |
| name | string | The application name |
| Application Type | string | The application type |
| queue | string | The queue the application was submitted to |
| state | string | The application state according to the ResourceManager - valid values are members of the YarnApplicationState enum: NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED |
| finalStatus | string | The final status of the application if finished - reported by the application itself - valid values are: UNDEFINED, SUCCEEDED, FAILED, KILLED |
| progress | float | The progress of the application as a percent |
| trackingUI | string | Where the tracking url is currently pointing - History (for history server) or ApplicationMaster |
| trackingUrl | string | The web URL that can be used to track the application |
| diagnostics | string | Detailed diagnostics information |
| clusterId | long | The cluster id |
| startedTime | long | The time in which application started (in ms since epoch) |
| finishedTime | long | The time in which the application finished (in ms since epoch) |
| elapsedTime | long | The elapsed time since the application started (in ms) |
| amContainerLogs | string | The URL of the application master container logs |
| amHostHttpAddress | string | The nodes http address of the application master |
| allocatedMB | int | The sum of memory in MB allocated to the application's running containers |
| allocatedVCores | int | The sum of virtual cores allocated to the application's running containers |
| runningContainers | int | The number of containers currently running for the application |
| memorySeconds | long | The amount of memory the application has allocated (megabyte-seconds) |
| vcoreSeconds | long | The amount of CPU resources the application has allocated (virtual core-seconds) |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1326821518301_0005
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"app" : {
"finishedTime" : 1326824991300,
"amContainerLogs" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001",
"trackingUI" : "History",
"state" : "FINISHED",
"user" : "user1",
"id" : "application_1326821518301_0005",
"clusterId" : 1326821518301,
"finalStatus" : "SUCCEEDED",
"amHostHttpAddress" : "host.domain.com:8042",
"progress" : 100,
"name" : "Sleep job",
"applicationType" : "Yarn",
"startedTime" : 1326824544552,
"elapsedTime" : 446748,
"diagnostics" : "",
"trackingUrl" : "http://host.domain.com:8088/proxy/application_1326821518301_0005/jobhistory/job/job_1326821518301_5_5",
"queue" : "a1",
"memorySeconds" : 151730,
"vcoreSeconds" : 103
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1326821518301_0005
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 847
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<app>
<id>application_1326821518301_0005</id>
<user>user1</user>
<name>Sleep job</name>
<queue>a1</queue>
<state>FINISHED</state>
<finalStatus>SUCCEEDED</finalStatus>
<progress>100.0</progress>
<trackingUI>History</trackingUI>
<trackingUrl>http://host.domain.com:8088/proxy/application_1326821518301_0005/jobhistory/job/job_1326821518301_5_5</trackingUrl>
<diagnostics/>
<clusterId>1326821518301</clusterId>
<startedTime>1326824544552</startedTime>
<finishedTime>1326824991300</finishedTime>
<elapsedTime>446748</elapsedTime>
<amContainerLogs>http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001</amContainerLogs>
<amHostHttpAddress>host.domain.com:8042</amHostHttpAddress>
<memorySeconds>151730</memorySeconds>
<vcoreSeconds>103</vcoreSeconds>
</app>
Cluster Application Attempts API
With the application attempts API, you can obtain a collection of resources that represent an application attempt. When you run a GET operation on this resource, you obtain a collection of App Attempt Objects.
URI
* http://<rm http address:port>/ws/v1/cluster/apps/{appid}/appattempts
HTTP Operations Supported
* GET
Query Parameters Supported
None
Elements of the appAttempts object
When you make a request for the list of app attempts, the information will be returned as an array of app attempt objects.
appAttempts:
| Item | Data Type | Description |
|---|---|---|
| appAttempt | array of app attempt objects(JSON)/zero or more app attempt objects(XML) | The collection of app attempt objects |
Elements of the appAttempt object
| Item | Data Type | Description |
|---|---|---|
| id | string | The app attempt id |
| nodeId | string | The node id of the node the attempt ran on |
| nodeHttpAddress | string | The node http address of the node the attempt ran on |
| logsLink | string | The http link to the app attempt logs |
| containerId | string | The id of the container for the app attempt |
| startTime | long | The start time of the attempt (in ms since epoch) |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1326821518301_0005/appattempts
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"appAttempts" : {
"appAttempt" : [
{
"nodeId" : "host.domain.com:8041",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381444693,
"id" : 1,
"logsLink" : "http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1",
"containerId" : "container_1326821518301_0005_01_000001"
}
]
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1326821518301_0005/appattempts
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 575
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appAttempts>
<appttempt>
<nodeHttpAddress>host.domain.com:8042</nodeHttpAddress>
<nodeId>host.domain.com:8041</nodeId>
<id>1</id>
<startTime>1326381444693</startTime>
<containerId>container_1326821518301_0005_01_000001</containerId>
<logsLink>http://host.domain.com:8042/node/containerlogs/container_1326821518301_0005_01_000001/user1</logsLink>
</appAttempt>
</appAttempts>
Cluster Nodes API
With the Nodes API, you can obtain a collection of resources, each of which represents a node. When you run a GET operation on this resource, you obtain a collection of Node Objects.
URI
* http://<rm http address:port>/ws/v1/cluster/nodes
HTTP Operations Supported
* GET
Query Parameters Supported
* state - the state of the node
* healthy - true or false
Elements of the nodes object
When you make a request for the list of nodes, the information will be returned as a collection of node objects. See also Node API for syntax of the node object.
| Item | Data Type | Description |
|---|---|---|
| node | array of node objects(JSON)/zero or more node objects(XML) | A collection of node objects |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/nodes
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"nodes":
{
"node":
[
{
"rack":"\/default-rack",
"state":"NEW",
"id":"h2:1235",
"nodeHostName":"h2",
"nodeHTTPAddress":"h2:2",
"healthStatus":"Healthy",
"lastHealthUpdate":1324056895432,
"healthReport":"Healthy",
"numContainers":0,
"usedMemoryMB":0,
"availMemoryMB":8192,
"usedVirtualCores":0,
"availableVirtualCores":8
},
{
"rack":"\/default-rack",
"state":"NEW",
"id":"h1:1234",
"nodeHostName":"h1",
"nodeHTTPAddress":"h1:2",
"healthStatus":"Healthy",
"lastHealthUpdate":1324056895092,
"healthReport":"Healthy",
"numContainers":0,
"usedMemoryMB":0,
"availMemoryMB":8192,
"usedVirtualCores":0,
"availableVirtualCores":8
}
]
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/nodes
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 1104
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<nodes>
<node>
<rack>/default-rack</rack>
<state>RUNNING</state>
<id>h2:1234</id>
<nodeHostName>h2</nodeHostName>
<nodeHTTPAddress>h2:2</nodeHTTPAddress>
<healthStatus>Healthy</healthStatus>
<lastHealthUpdate>1324333268447</lastHealthUpdate>
<healthReport>Healthy</healthReport>
<numContainers>0</numContainers>
<usedMemoryMB>0</usedMemoryMB>
<availMemoryMB>5120</availMemoryMB>
<usedVirtualCores>0</usedVirtualCores>
<availableVirtualCores>8</availableVirtualCores>
</node>
<node>
<rack>/default-rack</rack>
<state>RUNNING</state>
<id>h1:1234</id>
<nodeHostName>h1</nodeHostName>
<nodeHTTPAddress>h1:2</nodeHTTPAddress>
<healthStatus>Healthy</healthStatus>
<lastHealthUpdate>1324333268447</lastHealthUpdate>
<healthReport>Healthy</healthReport>
<numContainers>0</numContainers>
<usedMemoryMB>0</usedMemoryMB>
<availMemoryMB>5120</availMemoryMB>
<usedVirtualCores>0</usedVirtualCores>
<availableVirtualCores>8</availableVirtualCores>
</node>
</nodes>
Cluster Node API
A node resource contains information about a node in the cluster.
URI
Use the following URI to obtain a Node Object, from a node identified by the nodeid value.
* http://<rm http address:port>/ws/v1/cluster/nodes/{nodeid}
HTTP Operations Supported
* GET
Query Parameters Supported
None
Elements of the node object
| Item | Data Type | Description |
|---|---|---|
| rack | string | The rack location of this node |
| state | string | State of the node - valid values are: NEW, RUNNING, UNHEALTHY, DECOMMISSIONED, LOST, REBOOTED |
| id | string | The node id |
| nodeHostName | string | The host name of the node |
| nodeHTTPAddress | string | The nodes HTTP address |
| healthStatus | string | The health status of the node - Healthy or Unhealthy |
| healthReport | string | A detailed health report |
| lastHealthUpdate | long | The last time the node reported its health (in ms since epoch) |
| usedMemoryMB | long | The total amount of memory currently used on the node (in MB) |
| availMemoryMB | long | The total amount of memory currently available on the node (in MB) |
| usedVirtualCores | long | The total number of vCores currently used on the node |
| availableVirtualCores | long | The total number of vCores available on the node |
| numContainers | int | The total number of containers currently running on the node |
Response Examples
JSON response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/nodes/h2:1235
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"node":
{
"rack":"\/default-rack",
"state":"NEW",
"id":"h2:1235",
"nodeHostName":"h2",
"nodeHTTPAddress":"h2:2",
"healthStatus":"Healthy",
"lastHealthUpdate":1324056895432,
"healthReport":"Healthy",
"numContainers":0,
"usedMemoryMB":0,
"availMemoryMB":5120,
"usedVirtualCores":0,
"availableVirtualCores":8
}
}
XML response
HTTP Request:
GET http://<rm http address:port>/ws/v1/cluster/node/h2:1235
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 552
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<node>
<rack>/default-rack</rack>
<state>NEW</state>
<id>h2:1235</id>
<nodeHostName>h2</nodeHostName>
<nodeHTTPAddress>h2:2</nodeHTTPAddress>
<healthStatus>Healthy</healthStatus>
<lastHealthUpdate>1324333268447</lastHealthUpdate>
<healthReport>Healthy</healthReport>
<numContainers>0</numContainers>
<usedMemoryMB>0</usedMemoryMB>
<availMemoryMB>5120</availMemoryMB>
<usedVirtualCores>0</usedVirtualCores>
<availableVirtualCores>5120</availableVirtualCores>
</node>
Cluster Writeable APIs
The setions below refer to APIs which allow to create and modify applications. These APIs are currently in alpha and may change in the future.
Cluster New Application API
With the New Application API, you can obtain an application-id which can then be used as part of the Cluster Submit Applications API to submit applications. The response also includes the maximum resource capabilities available on the cluster.
This feature is currently in the alpha stage and may change in the future.
URI
* http://<rm http address:port>/ws/v1/cluster/apps/new-application
HTTP Operations Supported
* POST
Query Parameters Supported
None
Elements of the NewApplication object
The NewApplication response contains the following elements:
| Item | Data Type | Description |
|---|---|---|
| application-id | string | The newly created application id |
| maximum-resource-capabilities | object | The maximum resource capabilities available on this cluster |
The maximum-resource-capabilites object contains the following elements:
| Item | Data Type | Description |
|---|---|---|
| memory | int | The maxiumim memory available for a container |
| vCores | int | The maximum number of cores available for a container |
Response Examples
JSON response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/apps/new-application
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"application-id":"application_1404198295326_0003",
"maximum-resource-capability":
{
"memory":8192,
"vCores":32
}
}
XML response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/apps/new-application
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 248
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<NewApplication>
<application-id>application_1404198295326_0003</application-id>
<maximum-resource-capability>
<memory>8192</memory>
<vCores>32</vCores>
</maximum-resource-capability>
</NewApplication>
Cluster Applications API(Submit Application)
The Submit Applications API can be used to submit applications. In case of submitting applications, you must first obtain an application-id using the Cluster New Application API. The application-id must be part of the request body. The response contains a URL to the application page which can be used to track the state and progress of your application.
URI
* http://<rm http address:port>/ws/v1/cluster/apps
HTTP Operations Supported
* POST
POST Response Examples
POST requests can be used to submit apps to the ResourceManager. As mentioned above, an application-id must be obtained first. Successful submissions result in a 202 response code and a Location header specifying where to get information about the app. Please note that in order to submit an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response.
Please note that this feature is currently in the alpha stage and may change in the future.
Elements of the POST request object
| Item | Data Type | Description |
|---|---|---|
| application-id | string | The application id |
| application-name | string | The application name |
| queue | string | The name of the queue to which the application should be submitted |
| priority | int | The priority of the application |
| am-container-spec | object | The application master container launch context, described below |
| unmanaged-AM | boolean | Is the application using an unmanaged application master |
| max-app-attempts | int | The max number of attempts for this application |
| resource | object | The resources the application master requires, described below |
| application-type | string | The application type(MapReduce, Pig, Hive, etc) |
| keep-containers-across-application-attempts | boolean | Should YARN keep the containers used by this application instead of destroying them |
| application-tags | object | List of application tags, please see the request examples on how to speciy the tags |
Elements of the am-container-spec object
The am-container-spec object should be used to provide the container launch context for the application master.
| Item | Data Type | Description |
|---|---|---|
| local-resources | object | Object describing the resources that need to be localized, described below |
| environment | object | Environment variables for your containers, specified as key value pairs |
| commands | object | The commands for launching your container, in the order in which they should be executed |
| service-data | object | Application specific service data; key is the name of the auxiliary servce, value is base-64 encoding of the data you wish to pass |
| credentials | object | The credentials required for your application to run, described below |
| application-acls | objec | ACLs for your application; the key can be "VIEW_APP" or "MODIFY_APP", the value is the list of users with the permissions |
Elements of the local-resources object
The object is a collection of key-value pairs. They key is an identifier for the resources to be localized and the value is the details of the resource. The elements of the value are described below:
| Item | Data Type | Description |
|---|---|---|
| resource | string | Location of the resource to be localized |
| type | string | Type of the resource; options are "ARCHIVE", "FILE", and "PATTERN" |
| visibility | string | Visibility the resource to be localized; options are "PUBLIC", "PRIVATE", and "APPLICATION" |
| size | long | Size of the resource to be localized |
| timestamp | long | Timestamp of the resource to be localized |
Elements of the credentials object
The credentials object should be used to pass data required for the application to authenticate itself such as delegation-tokens and secrets.
| Item | Data Type | Description |
|---|---|---|
| tokens | object | Tokens that you wish to pass to your application, specified as key-value pairs. The key is an identifier for the token and the value is the token(which should be obtained using the respective web-services) |
| secrets | object | Secrets that you wish to use in your application, specified as key-value pairs. They key is an identifier and the value is the base-64 encoding of the secret |
Elements of the POST request body resource object
| Item | Data Type | Description |
|---|---|---|
| memory | int | Memory required for each container |
| vCores | int | Virtual cores required for each container |
JSON response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/apps
Accept: application/json
Content-Type: application/json
{
"application-id":"application_1404203615263_0001",
"application-name":"test",
"am-container-spec":
{
"local-resources":
{
"entry":
[
{
"key":"AppMaster.jar",
"value":
{
"resource":"hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar",
"type":"FILE",
"visibility":"APPLICATION",
"size": 43004,
"timestamp": 1405452071209
}
}
]
},
"commands":
{
"command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr"
},
"environment":
{
"entry":
[
{
"key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP",
"value": "1405459400754"
},
{
"key": "CLASSPATH",
"value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
},
{
"key": "DISTRIBUTEDSHELLSCRIPTLEN",
"value": "6"
},
{
"key": "DISTRIBUTEDSHELLSCRIPTLOCATION",
"value": "hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands"
}
]
}
},
"unmanaged-AM":false,
"max-app-attempts":2,
"resource":
{
"memory":1024,
"vCores":1
},
"application-type":"YARN",
"keep-containers-across-application-attempts":false
}
Response Header:
HTTP/1.1 202
Transfer-Encoding: chunked
Location: http://<rm http address:port>/ws/v1/cluster/apps/application_1404203615263_0001
Content-Type: application/json
Server: Jetty(6.1.26)
Response Body:
No response body
XML response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/apps
Accept: application/xml
Content-Type: application/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application-submission-context>
<application-id>application_1404204891930_0002</application-id>
<application-name>test</application-name>
<queue>testqueue</queue>
<priority>3</priority>
<am-container-spec>
<local-resources>
<entry>
<key>example</key>
<value>
<resource>hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar</resource>
<type>FILE</type>
<visibility>APPLICATION</visibility>
<size>43004</size>
<timestamp>1405452071209</timestamp>
</value>
</entry>
</local-resources>
<environment>
<entry>
<key>DISTRIBUTEDSHELLSCRIPTTIMESTAMP</key>
<value>1405459400754</value>
</entry>
<entry>
<key>CLASSPATH</key>
<value>{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties</value>
</entry>
<entry>
<key>DISTRIBUTEDSHELLSCRIPTLEN</key>
<value>6</value>
</entry>
<entry>
<key>DISTRIBUTEDSHELLSCRIPTLOCATION</key>
<value>hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands</value>
</entry>
</environment>
<commands>
<command>{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr</command>
</commands>
<service-data>
<entry>
<key>test</key>
<value>dmFsdWUxMg</value>
</entry>
</service-data>
<credentials>
<tokens/>
<secrets>
<entry>
<key>secret1</key>
<value>c2VjcmV0MQ</value>
</entry>
</secrets>
</credentials>
<application-acls>
<entry>
<key>VIEW_APP</key>
<value>testuser3, testuser4</value>
</entry>
<entry>
<key>MODIFY_APP</key>
<value>testuser1, testuser2</value>
</entry>
</application-acls>
</am-container-spec>
<unmanaged-AM>false</unmanaged-AM>
<max-app-attempts>2</max-app-attempts>
<resource>
<memory>1024</memory>
<vCores>1</vCores>
</resource>
<application-type>YARN</application-type>
<keep-containers-across-application-attempts>false</keep-containers-across-application-attempts>
<application-tags>
<tag>tag 2</tag>
<tag>tag1</tag>
</application-tags>
</application-submission-context>
Response Header:
HTTP/1.1 202
Transfer-Encoding: chunked
Location: http://<rm http address:port>/ws/v1/cluster/apps/application_1404204891930_0002
Content-Type: application/xml
Server: Jetty(6.1.26)
Response Body:
No response body
Cluster Application State API
With the application state API, you can query the state of a submitted app as well kill a running app by modifying the state of a running app using a PUT request with the state set to "KILLED". To perform the PUT operation, authentication has to be setup for the RM web services. In addition, you must be authorized to kill the app. Currently you can only change the state to "KILLED"; an attempt to change the state to any other results in a 400 error response. Examples of the unauthorized and bad request errors are below. When you carry out a successful PUT, the iniital response may be a 202. You can confirm that the app is killed by repeating the PUT request until you get a 200, querying the state using the GET method or querying for app information and checking the state. In the examples below, we repeat the PUT request and get a 200 response.
Please note that in order to kill an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response.
This feature is currently in the alpha stage and may change in the future.
URI
* http://<rm http address:port>/ws/v1/cluster/apps/{appid}/state
HTTP Operations Supported
* GET
* PUT
Query Parameters Supported
None
Elements of appstate object
When you make a request for the state of an app, the information returned has the following fields
| Item | Data Type | Description |
|---|---|---|
| state | string | The application state - can be one of "NEW", "NEW_SAVING", "SUBMITTED", "ACCEPTED", "RUNNING", "FINISHED", "FAILED", "KILLED" |
Response Examples
JSON responses
HTTP Request
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"state":"ACCEPTED"
}
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
{
"state":"KILLED"
}
Response Header:
HTTP/1.1 202 Accepted
Content-Type: application/json
Transfer-Encoding: chunked
Location: http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003
Server: Jetty(6.1.26)
Response Body:
{
"state":"ACCEPTED"
}
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
{
"state":"KILLED"
}
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"state":"KILLED"
}
XML responses
HTTP Request
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 99
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>ACCEPTED</state>
</appstate>
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>KILLED</state>
</appstate>
Response Header:
HTTP/1.1 202 Accepted
Content-Type: application/xml
Content-Length: 794
Location: http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>ACCEPTED</state>
</appstate>
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>KILLED</state>
</appstate>
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 917
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>KILLED</state>
</appstate>
Unauthorized Error Response
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>KILLED</state>
</appstate>
Response Header:
HTTP/1.1 403 Unauthorized
Server: Jetty(6.1.26)
Bad Request Error Response
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
Request Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appstate>
<state>RUNNING</state>
</appstate>
Response Header:
HTTP/1.1 400
Content-Length: 295
Content-Type: application/xml
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<RemoteException>
<exception>BadRequestException</exception>
<message>java.lang.Exception: Only 'KILLED' is allowed as a target state.</message>
<javaClassName>org.apache.hadoop.yarn.webapp.BadRequestException</javaClassName>
</RemoteException>
Cluster Application Queue API
With the application queue API, you can query the queue of a submitted app as well move a running app to another queue using a PUT request specifying the target queue. To perform the PUT operation, authentication has to be setup for the RM web services. In addition, you must be authorized to move the app. Currently you can only move the app if you're using the Capacity scheduler or the Fair scheduler.
Please note that in order to move an app, you must have an authentication filter setup for the HTTP interface. The functionality requires that a username is set in the HttpServletRequest. If no filter is setup, the response will be an "UNAUTHORIZED" response.
This feature is currently in the alpha stage and may change in the future.
URI
* http://<rm http address:port>/ws/v1/cluster/apps/{appid}/queue
HTTP Operations Supported
* GET
* PUT
Query Parameters Supported
None
Elements of appqueue object
When you make a request for the state of an app, the information returned has the following fields
| Item | Data Type | Description |
|---|---|---|
| queue | string | The application queue |
Response Examples
JSON responses
HTTP Request
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/queue
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"queue":"default"
}
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/queue
Request Body:
{
"queue":"test"
}
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"queue":"test"
}
XML responses
HTTP Request
GET http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/queue
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 98
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appqueue>
<queue>default</queue>
</appqueue>
HTTP Request
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/queue
Request Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appqueue>
<queue>test</queue>
</appqueue>
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 95
Server: Jetty(6.1.26)
Response Body:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<appqueue>
<queue>test</queue>
</appqueue>
Cluster Delegation Tokens API
The Delegation Tokens API can be used to create, renew and cancel YARN ResourceManager delegation tokens. All delegation token requests must be carried out on a Kerberos authenticated connection(using SPNEGO). Carrying out operations on a non-kerberos connection will result in a FORBIDDEN response. In case of renewing a token, only the renewer specified when creating the token can renew the token. Other users(including the owner) are forbidden from renewing tokens. It should be noted that when cancelling or renewing a token, the token to be cancelled or renewed is specified by setting a header.
This feature is currently in the alpha stage and may change in the future.
URI
Use the following URI to create and cancel delegation tokens.
* http://<rm http address:port>/ws/v1/cluster/delegation-token
Use the following URI to renew delegation tokens.
* http://<rm http address:port>/ws/v1/cluster/delegation-token/expiration
HTTP Operations Supported
* POST
* DELETE
Query Parameters Supported
None
Elements of the delegation-token object
The response from the delegation tokens API contains one of the fields listed below.
| Item | Data Type | Description |
|---|---|---|
| token | string | The delegation token |
| renewer | string | The user who is allowed to renew the delegation token |
| owner | string | The owner of the delegation token |
| kind | string | The kind of delegation token |
| expiration-time | long | The expiration time of the token |
| max-validity | long | The maximum validity of the token |
Response Examples
Creating a token
JSON response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/delegation-token
Accept: application/json
Content-Type: application/json
{
"renewer" : "test-renewer"
}
Response Header
HTTP/1.1 200 OK
WWW-Authenticate: Negotiate ...
Date: Sat, 28 Jun 2014 18:08:11 GMT
Server: Jetty(6.1.26)
Set-Cookie: ...
Content-Type: application/json
Response body
{
"token":"MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUckiEZpigFHSJTKaQECFN9EMM9BzfPoDxu572EVUpzqhnSGE1JNX0RFTEVHQVRJT05fVE9LRU4A",
"renewer":"test-renewer",
"owner":"client@EXAMPLE.COM",
"kind":"RM_DELEGATION_TOKEN",
"expiration-time":1405153616489,
"max-validity":1405672016489
}
XML response
HTTP Request
POST http://<rm http address:port>/ws/v1/cluster/delegation-token
Accept: application/xml
Content-Type: application/xml
<delegation-token>
<renewer>test-renewer</renewer>
</delegation-token>
Response Header
HTTP/1.1 200 OK
WWW-Authenticate: Negotiate ...
Date: Sat, 28 Jun 2014 18:08:11 GMT
Content-Length: 423
Server: Jetty(6.1.26)
Set-Cookie: ...
Content-Type: application/xml
Response Body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<delegation-token> <token>MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUckgZ8yigFHSI4jMgcCFDTG8X6XFFn2udQngzSXQL8vWaKIE1JNX0RFTEVHQVRJT05fVE9LRU4A</token>
<renewer>test-renewer</renewer>
<owner>client@EXAMPLE.COM</owner>
<kind>RM_DELEGATION_TOKEN</kind>
<expiration-time>1405153180466</expiration-time>
<max-validity>1405671580466</max-validity>
</delegation-token>
Renewing a token
JSON response
HTTP Request:
POST http://<rm http address:port>/ws/v1/cluster/delegation-token/expiration
Accept: application/json
Hadoop-YARN-RM-Delegation-Token: MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUbjqcHHigFHB7ZFxwQCFKWD3znCkDSy6SQIjRCLDydxbxvgE1JNX0RFTEVHQVRJT05fVE9LRU4A
Content-Type: application/json
Response Header
HTTP/1.1 200 OK
WWW-Authenticate: Negotiate ...
Date: Sat, 28 Jun 2014 18:08:11 GMT
Server: Jetty(6.1.26)
Set-Cookie: ...
Content-Type: application/json
Response body
{
"expiration-time":1404112520402
}
XML response
HTTP Request
POST http://<rm http address:port>/ws/v1/cluster/delegation-token/expiration
Accept: application/xml
Content-Type: application/xml
Hadoop-YARN-RM-Delegation-Token: MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUbjqcHHigFHB7ZFxwQCFKWD3znCkDSy6SQIjRCLDydxbxvgE1JNX0RFTEVHQVRJT05fVE9LRU4A
Response Header
HTTP/1.1 200 OK
WWW-Authenticate: Negotiate ...
Date: Sat, 28 Jun 2014 18:08:11 GMT
Content-Length: 423
Server: Jetty(6.1.26)
Set-Cookie: ...
Content-Type: application/xml
Response Body
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<delegation-token>
<expiration-time>1404112520402</expiration-time>
</delegation-token>
Cancelling a token
HTTP Request
DELETE http://<rm http address:port>/ws/v1/cluster/delegation-token
Hadoop-YARN-RM-Delegation-Token: MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUbjqcHHigFHB7ZFxwQCFKWD3znCkDSy6SQIjRCLDydxbxvgE1JNX0RFTEVHQVRJT05fVE9LRU4A
Accept: application/xml
Response Header
HTTP/1.1 200 OK
WWW-Authenticate: Negotiate ...
Date: Sun, 29 Jun 2014 07:25:18 GMT
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Set-Cookie: ...
Content-Type: application/xml
No response body.
Authentication using delegation tokens
This feature is in the alpha mode and may change in the future.
You can use delegation tokens to authenticate yourself when using YARN RM webservices. However, this requires setting the right configurations. The conditions for this are:
-
Hadoop is setup in secure mode with the authentication type set to kerberos.
-
Hadoop HTTP authentication is setup with the authentication type set to kerberos
Once setup, delegation tokens can be fetched using the web services listed above and used as shown in an example below:
PUT http://<rm http address:port>/ws/v1/cluster/apps/application_1399397633663_0003/state
X-Hadoop-Delegation-Token: MgASY2xpZW50QEVYQU1QTEUuQ09NDHRlc3QtcmVuZXdlcgCKAUbjqcHHigFHB7ZFxwQCFKWD3znCkDSy6SQIjRCLDydxbxvgE1JNX0RFTEVHQVRJT05fVE9LRU4A
Content-Type: application/json; charset=UTF8
{
"state":"KILLED"
}