YARN Timeline Server ==================== * [Overview](#Overview) * [Current Status](#Current_Status) * [Basic Configuration](#Basic_Configuration) * [Advanced Configuration](#Advanced_Configuration) * [Generic-data related Configuration](#Generic-data_related_Configuration) * [Per-framework-date related Configuration](#Per-framework-date_related_Configuration) * [Running Timeline server](#Running_Timeline_server) * [Accessing generic-data via command-line](#Accessing_generic-data_via_command-line) * [Publishing of per-framework data by applications](#Publishing_of_per-framework_data_by_applications) Overview -------- Storage and retrieval of applications' current as well as historic information in a generic fashion is solved in YARN through the Timeline Server (previously also called Generic Application History Server). This serves two responsibilities: * Generic information about completed applications Generic information includes application level data like queue-name, user information etc in the ApplicationSubmissionContext, list of application-attempts that ran for an application, information about each application-attempt, list of containers run under each application-attempt, and information about each container. Generic data is stored by ResourceManager to a history-store (default implementation on a file-system) and used by the web-UI to display information about completed applications. * Per-framework information of running and completed applications Per-framework information is completely specific to an application or framework. For example, Hadoop MapReduce framework can include pieces of information like number of map tasks, reduce tasks, counters etc. Application developers can publish the specific information to the Timeline server via TimelineClient from within a client, the ApplicationMaster and/or the application's containers. This information is then queryable via REST APIs for rendering by application/framework specific UIs. Current Status -------------- Timeline sever is a work in progress. The basic storage and retrieval of information, both generic and framework specific, are in place. Timeline server doesn't work in secure mode yet. The generic information and the per-framework information are today collected and presented separately and thus are not integrated well together. Finally, the per-framework information is only available via RESTful APIs, using JSON type content - ability to install framework specific UIs in YARN isn't supported yet. Basic Configuration ------------------- Users need to configure the Timeline server before starting it. The simplest configuration you should add in `yarn-site.xml` is to set the hostname of the Timeline server. ```xml The hostname of the Timeline service web application. yarn.timeline-service.hostname 0.0.0.0 ``` Advanced Configuration ---------------------- In addition to the hostname, admins can also configure whether the service is enabled or not, the ports of the RPC and the web interfaces, and the number of RPC handler threads. ```xml Address for the Timeline server to start the RPC server. yarn.timeline-service.address ${yarn.timeline-service.hostname}:10200 The http address of the Timeline service web application. yarn.timeline-service.webapp.address ${yarn.timeline-service.hostname}:8188 The https address of the Timeline service web application. yarn.timeline-service.webapp.https.address ${yarn.timeline-service.hostname}:8190 Handler thread count to serve the client RPC requests. yarn.timeline-service.handler-thread-count 10 Enables cross-origin support (CORS) for web services where cross-origin web response headers are needed. For example, javascript making a web services request to the timeline server. yarn.timeline-service.http-cross-origin.enabled false Comma separated list of origins that are allowed for web services needing cross-origin (CORS) support. Wildcards (*) and patterns allowed yarn.timeline-service.http-cross-origin.allowed-origins * Comma separated list of methods that are allowed for web services needing cross-origin (CORS) support. yarn.timeline-service.http-cross-origin.allowed-methods GET,POST,HEAD Comma separated list of headers that are allowed for web services needing cross-origin (CORS) support. yarn.timeline-service.http-cross-origin.allowed-headers X-Requested-With,Content-Type,Accept,Origin The number of seconds a pre-flighted request can be cached for web services needing cross-origin (CORS) support. yarn.timeline-service.http-cross-origin.max-age 1800 ``` Generic-data related Configuration ---------------------------------- Users can specify whether the generic data collection is enabled or not, and also choose the storage-implementation class for the generic data. There are more configurations related to generic data collection, and users can refer to `yarn-default.xml` for all of them. ```xml Indicate to ResourceManager as well as clients whether history-service is enabled or not. If enabled, ResourceManager starts recording historical data that Timelien service can consume. Similarly, clients can redirect to the history service when applications finish if this is enabled. yarn.timeline-service.generic-application-history.enabled false Store class name for history store, defaulting to file system store yarn.timeline-service.generic-application-history.store-class org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore ``` Per-framework-date related Configuration ---------------------------------------- Users can specify whether per-framework data service is enabled or not, choose the store implementation for the per-framework data, and tune the retention of the per-framework data. There are more configurations related to per-framework data service, and users can refer to `yarn-default.xml` for all of them. ```xml Indicate to clients whether Timeline service is enabled or not. If enabled, the TimelineClient library used by end-users will post entities and events to the Timeline server. yarn.timeline-service.enabled true Store class name for timeline store. yarn.timeline-service.store-class org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore Enable age off of timeline store data. yarn.timeline-service.ttl-enable true Time to live for timeline store data in milliseconds. yarn.timeline-service.ttl-ms 604800000 ``` Running Timeline server ----------------------- Assuming all the aforementioned configurations are set properly, admins can start the Timeline server/history service with the following command: $ yarn timelineserver Or users can start the Timeline server / history service as a daemon: $ yarn --daemon start timelineserver Accessing generic-data via command-line --------------------------------------- Users can access applications' generic historic data via the command line as below. Note that the same commands are usable to obtain the corresponding information about running applications. ``` $ yarn application -status $ yarn applicationattempt -list $ yarn applicationattempt -status $ yarn container -list $ yarn container -status ``` Publishing of per-framework data by applications ------------------------------------------------ Developers can define what information they want to record for their applications by composing `TimelineEntity` and `TimelineEvent` objects, and put the entities and events to the Timeline server via `TimelineClient`. Following is an example: ```java // Create and start the Timeline client TimelineClient client = TimelineClient.createTimelineClient(); client.init(conf); client.start(); TimelineEntity entity = null; // Compose the entity try { TimelinePutResponse response = client.putEntities(entity); } catch (IOException e) { // Handle the exception } catch (YarnException e) { // Handle the exception } // Stop the Timeline client client.stop(); ```