Get a report of the given Application.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

If the user does not have VIEW_APP access then the following fields in the report will be set to stubbed values:

  • host - set to "N/A"
  • RPC port - set to -1
  • client token - set to "N/A"
  • diagnostics - set to "N/A"
  • tracking URL - set to "N/A"
  • original tracking URL - set to "N/A"
  • resource usage report - all values are -1

@param appId {@link ApplicationId} of the application that needs a report @return application report @throws YarnException @throws IOException]]> Get a report (ApplicationReport) of all Applications in the cluster.

If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in {@link #getApplicationReport(ApplicationId)}.

@return a list of reports for all applications @throws YarnException @throws IOException]]> Get a report of the given ApplicationAttempt.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

@param applicationAttemptId {@link ApplicationAttemptId} of the application attempt that needs a report @return application attempt report @throws YarnException @throws {@link ApplicationAttemptNotFoundException} if application attempt not found @throws IOException]]> Get a report of all (ApplicationAttempts) of Application in the cluster.

@param applicationId @return a list of reports for all application attempts for specified application @throws YarnException @throws IOException]]> Get a report of the given Container.

In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

@param containerId {@link ContainerId} of the container that needs a report @return container report @throws YarnException @throws {@link ContainerNotFoundException} if container not found @throws IOException]]> Get a report of all (Containers) of ApplicationAttempt in the cluster.

@param applicationAttemptId @return a list of reports of all containers for specified application attempt @throws YarnException @throws IOException]]> {@code AMRMClient.createAMRMClientContainerRequest() } @return the newly create AMRMClient instance.]]> RegisterApplicationMasterResponse @throws YarnException @throws IOException]]> addContainerRequest are sent to the ResourceManager. New containers assigned to the master are retrieved. Status of completed containers and node health updates are also retrieved. This also doubles up as a heartbeat to the ResourceManager and must be made periodically. The call may not always return any new allocations of containers. App should not make concurrent allocate requests. May cause request loss.

Note : If the user has not removed container requests that have already been satisfied, then the re-register may end up sending the entire container requests to the RM (including matched requests). Which would mean the RM could end up giving it a lot of new allocated containers.

@param progressIndicator Indicates progress made by the master @return the response of the allocate request @throws YarnException @throws IOException]]> allocate @param req Resource request]]> ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added. @return Collection of request matching the parameters]]> AMRMClient. This cache must be shared with the {@link NMClient} used to manage containers for the AMRMClient

If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} singleton instance will be used. @param nmTokenCache the NM token cache to use.]]> AMRMClient. This cache must be shared with the {@link NMClient} used to manage containers for the AMRMClient.

If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} singleton instance will be used. @return the NM token cache.]]> check to return true for each 1000 ms. See also {@link #waitFor(com.google.common.base.Supplier, int)} and {@link #waitFor(com.google.common.base.Supplier, int, int)} @param check]]> check to return true for each checkEveryMillis ms. See also {@link #waitFor(com.google.common.base.Supplier, int, int)} @param check user defined checker @param checkEveryMillis interval to call check]]> check to return true for each checkEveryMillis ms. In the main loop, this method will log the message "waiting in main loop" for each logInterval times iteration to confirm the thread is alive. @param check user defined checker @param checkEveryMillis interval to call check @param logInterval interval to log for each]]>

  • Within a priority, all container requests must have the same value for locality relaxation. Either enabled or disabled.
  • If locality relaxation is disabled, then across requests, locations at different network levels may not be specified. E.g. its invalid to make a request for a specific node and another request for a specific rack.
  • If locality relaxation is disabled, then only within the same request, a node and its rack may be specified together. This allows for a specific rack with a preference for a specific node within that rack.
  • To re-enable locality relaxation at a given priority, all pending requests with locality relaxation disabled must be first removed. Then they can be added back with locality relaxation enabled. All getters return immutable values.]]>     AMRMClient.ContainerRequest in an invalid way.]]> Start an allocated container.

    The ApplicationMaster or other applications that use the client must provide the details of the allocated container, including the Id, the assigned node's Id and the token via {@link Container}. In addition, the AM needs to provide the {@link ContainerLaunchContext} as well.

    @param container the allocated container @param containerLaunchContext the context information needed by the NodeManager to launch the container @return a map between the auxiliary service names and their outputs @throws YarnException @throws IOException]]>     Stop an started container.

    @param containerId the Id of the started container @param nodeId the Id of the NodeManager @throws YarnException @throws IOException]]>     Query the status of a container.

    @param containerId the Id of the started container @param nodeId the Id of the NodeManager @return the status of a container @throws YarnException @throws IOException]]>     Set whether the containers that are started by this client, and are still running should be stopped when the client stops. By default, the feature should be enabled.

    However, containers will be stopped only when service is stopped. i.e. after {@link NMClient#stop()}. @param enabled whether the feature is enabled or not]]>     NMClient. This cache must be shared with the {@link AMRMClient} that requested the containers managed by this NMClient

    If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} singleton instance will be used. @param nmTokenCache the NM token cache to use.]]> NMClient. This cache must be shared with the {@link AMRMClient} that requested the containers managed by this NMClient

    If a NM token cache is not set, the {@link NMTokenCache#getSingleton()} singleton instance will be used. @return the NM token cache]]> By default Yarn client libraries {@link AMRMClient} and {@link NMClient} use {@link #getSingleton()} instance of the cache.

    • Using the singleton instance of the cache is appropriate when running a single ApplicationMaster in the same JVM.
    • When using the singleton, users don't need to do anything special, {@link AMRMClient} and {@link NMClient} are already set up to use the default singleton {@link NMTokenCache}

    If running multiple Application Masters in the same JVM, a different cache instance should be used for each Application Master.

    • If using the {@link AMRMClient} and the {@link NMClient}, setting up and using an instance cache is as follows: 

         NMTokenCache nmTokenCache = new NMTokenCache();
   AMRMClient rmClient = AMRMClient.createAMRMClient();
   NMClient nmClient = NMClient.createNMClient();
   nmClient.setNMTokenCache(nmTokenCache);
   ...
    • If using the {@link AMRMClientAsync} and the {@link NMClientAsync}, setting up and using an instance cache is as follows: 

         NMTokenCache nmTokenCache = new NMTokenCache();
   AMRMClient rmClient = AMRMClient.createAMRMClient();
   NMClient nmClient = NMClient.createNMClient();
   nmClient.setNMTokenCache(nmTokenCache);
   AMRMClientAsync rmClientAsync = new AMRMClientAsync(rmClient, 1000, [AMRM_CALLBACK]);
   NMClientAsync nmClientAsync = new NMClientAsync("nmClient", nmClient, [NM_CALLBACK]);
   ...
    • If using {@link ApplicationMasterProtocol} and {@link ContainerManagementProtocol} directly, setting up and using an instance cache is as follows: 

         NMTokenCache nmTokenCache = new NMTokenCache();
   ...
   ApplicationMasterProtocol amPro = ClientRMProxy.createRMProxy(conf, ApplicationMasterProtocol.class);
   ...
   AllocateRequest allocateRequest = ...
   ...
   AllocateResponse allocateResponse = rmClient.allocate(allocateRequest);
   for (NMToken token : allocateResponse.getNMTokens()) {
     nmTokenCache.setToken(token.getNodeId().toString(), token.getToken());
   }
   ...
   ContainerManagementProtocolProxy nmPro = ContainerManagementProtocolProxy(conf, nmTokenCache);
   ...
   nmPro.startContainer(container, containerContext);
   ...
    It is also possible to mix the usage of a client (AMRMClient or NMClient, or the async versions of them) with a protocol proxy ( ContainerManagementProtocolProxy or ApplicationMasterProtocol).]]>     Obtain a {@link YarnClientApplication} for a new application, which in turn contains the {@link ApplicationSubmissionContext} and {@link org.apache.hadoop.yarn.api.protocolrecords.GetNewApplicationResponse} objects.

    @return {@link YarnClientApplication} built for a new application @throws YarnException @throws IOException]]>     Submit a new application to YARN. It is a blocking call - it will not return {@link ApplicationId} until the submitted application is submitted successfully and accepted by the ResourceManager.

    Users should provide an {@link ApplicationId} as part of the parameter {@link ApplicationSubmissionContext} when submitting a new application, otherwise it will throw the {@link ApplicationIdNotProvidedException}.

    This internally calls {@link ApplicationClientProtocol#submitApplication (SubmitApplicationRequest)}, and after that, it internally invokes {@link ApplicationClientProtocol#getApplicationReport (GetApplicationReportRequest)} and waits till it can make sure that the application gets properly submitted. If RM fails over or RM restart happens before ResourceManager saves the application's state, {@link ApplicationClientProtocol #getApplicationReport(GetApplicationReportRequest)} will throw the {@link ApplicationNotFoundException}. This API automatically resubmits the application with the same {@link ApplicationSubmissionContext} when it catches the {@link ApplicationNotFoundException}

    @param appContext {@link ApplicationSubmissionContext} containing all the details needed to submit a new application @return {@link ApplicationId} of the accepted application @throws YarnException @throws IOException @see #createApplication()]]>     Kill an application identified by given ID.

    @param applicationId {@link ApplicationId} of the application that needs to be killed @throws YarnException in case of errors or if YARN rejects the request due to access-control restrictions. @throws IOException @see #getQueueAclsInfo()]]>     Get a report of the given Application.

    In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

    If the user does not have VIEW_APP access then the following fields in the report will be set to stubbed values:

    • host - set to "N/A"
    • RPC port - set to -1
    • client token - set to "N/A"
    • diagnostics - set to "N/A"
    • tracking URL - set to "N/A"
    • original tracking URL - set to "N/A"
    • resource usage report - all values are -1

    @param appId {@link ApplicationId} of the application that needs a report @return application report @throws YarnException @throws IOException]]>     The AMRM token is required for AM to RM scheduling operations. For managed Application Masters Yarn takes care of injecting it. For unmanaged Applications Masters, the token must be obtained via this method and set in the {@link org.apache.hadoop.security.UserGroupInformation} of the current user.

    The AMRM token will be returned only if all the following conditions are met:

    • the requester is the owner of the ApplicationMaster
      the application master is an unmanaged ApplicationMaster
      the application master is in ACCEPTED state
    • Else this method returns NULL. @param appId {@link ApplicationId} of the application to get the AMRM token @return the AMRM token if available @throws YarnException @throws IOException]]>     Get a report (ApplicationReport) of all Applications in the cluster.

    If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in {@link #getApplicationReport(ApplicationId)}.

    @return a list of reports of all running applications @throws YarnException @throws IOException]]>     Get a report (ApplicationReport) of Applications matching the given application types in the cluster.

    If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in {@link #getApplicationReport(ApplicationId)}.

    @param applicationTypes @return a list of reports of applications @throws YarnException @throws IOException]]>     Get a report (ApplicationReport) of Applications matching the given application states in the cluster.

    If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in {@link #getApplicationReport(ApplicationId)}.

    @param applicationStates @return a list of reports of applications @throws YarnException @throws IOException]]>     Get a report (ApplicationReport) of Applications matching the given application types and application states in the cluster.

    If the user does not have VIEW_APP access for an application then the corresponding report will be filtered as described in {@link #getApplicationReport(ApplicationId)}.

    @param applicationTypes @param applicationStates @return a list of reports of applications @throws YarnException @throws IOException]]>     Get metrics ({@link YarnClusterMetrics}) about the cluster.

    @return cluster metrics @throws YarnException @throws IOException]]>     Get a report of nodes ({@link NodeReport}) in the cluster.

    @param states The {@link NodeState}s to filter on. If no filter states are given, nodes in all states will be returned. @return A list of node reports @throws YarnException @throws IOException]]>     Get a delegation token so as to be able to talk to YARN using those tokens. @param renewer Address of the renewer who can renew these tokens when needed by securely talking to YARN. @return a delegation token ({@link Token}) that can be used to talk to YARN @throws YarnException @throws IOException]]> Get information ({@link QueueInfo}) about a given queue.

    @param queueName Name of the queue whose information is needed @return queue information @throws YarnException in case of errors or if YARN rejects the request due to access-control restrictions. @throws IOException]]>     Get information ({@link QueueInfo}) about all queues, recursively if there is a hierarchy

    @return a list of queue-information for all queues @throws YarnException @throws IOException]]>     Get information ({@link QueueInfo}) about top level queues.

    @return a list of queue-information for all the top-level queues @throws YarnException @throws IOException]]>     Get information ({@link QueueInfo}) about all the immediate children queues of the given queue

    @param parent Name of the queue whose child-queues' information is needed @return a list of queue-information for all queues who are direct children of the given parent queue. @throws YarnException @throws IOException]]>     Get information about acls for current user on all the existing queues.

    @return a list of queue acls ({@link QueueUserACLInfo}) for current user @throws YarnException @throws IOException]]>     Get a report of the given ApplicationAttempt.

    In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

    @param applicationAttemptId {@link ApplicationAttemptId} of the application attempt that needs a report @return application attempt report @throws YarnException @throws {@link ApplicationAttemptNotFoundException} if application attempt not found @throws IOException]]>     Get a report of all (ApplicationAttempts) of Application in the cluster.

    @param applicationId @return a list of reports for all application attempts for specified application. @throws YarnException @throws IOException]]>     Get a report of the given Container.

    In secure mode, YARN verifies access to the application, queue etc. before accepting the request.

    @param containerId {@link ContainerId} of the container that needs a report @return container report @throws YarnException @throws {@link ContainerNotFoundException} if container not found. @throws IOException]]>     Get a report of all (Containers) of ApplicationAttempt in the cluster.

    @param applicationAttemptId @return a list of reports of all containers for specified application attempts @throws YarnException @throws IOException]]>     Attempts to move the given application to the given queue.

    @param appId Application to move. @param queue Queue to place it in to. @throws YarnException @throws IOException]]>     The interface used by clients to submit a new reservation to the {@code ResourceManager}.

    The client packages all details of its request in a {@link ReservationSubmissionRequest} object. This contains information about the amount of capacity, temporal constraints, and gang needs. Furthermore, the reservation might be composed of multiple stages, with ordering dependencies among them.

    In order to respond, a new admission control component in the {@code ResourceManager} performs an analysis of the resources that have been committed over the period of time the user is requesting, verify that the user requests can be fulfilled, and that it respect a sharing policy (e.g., {@code CapacityOverTimePolicy}). Once it has positively determined that the ReservationRequest is satisfiable the {@code ResourceManager} answers with a {@link ReservationSubmissionResponse} that includes a {@link ReservationId}. Upon failure to find a valid allocation the response is an exception with the message detailing the reason of failure.

    The semantics guarantees that the {@link ReservationId} returned, corresponds to a valid reservation existing in the time-range request by the user. The amount of capacity dedicated to such reservation can vary overtime, depending of the allocation that has been determined. But it is guaranteed to satisfy all the constraint expressed by the user in the {@link ReservationDefinition}

    @param request request to submit a new Reservation @return response contains the {@link ReservationId} on accepting the submission @throws YarnException if the reservation cannot be created successfully @throws IOException]]>     The interface used by clients to update an existing Reservation. This is referred to as a re-negotiation process, in which a user that has previously submitted a Reservation.

    The allocation is attempted by virtually substituting all previous allocations related to this Reservation with new ones, that satisfy the new {@link ReservationDefinition}. Upon success the previous allocation is atomically substituted by the new one, and on failure (i.e., if the system cannot find a valid allocation for the updated request), the previous allocation remains valid.

    @param request to update an existing Reservation (the {@link ReservationUpdateRequest} should refer to an existing valid {@link ReservationId}) @return response empty on successfully updating the existing reservation @throws YarnException if the request is invalid or reservation cannot be updated successfully @throws IOException]]>     The interface used by clients to remove an existing Reservation.

    @param request to remove an existing Reservation (the {@link ReservationDeleteRequest} should refer to an existing valid {@link ReservationId}) @return response empty on successfully deleting the existing reservation @throws YarnException if the request is invalid or reservation cannot be deleted successfully @throws IOException]]>     The interface used by client to get node to labels mappings in existing cluster

    @return node to labels mappings @throws YarnException @throws IOException]]>     The interface used by client to get node labels in the cluster

    @return cluster node labels collection @throws YarnException @throws IOException]]>     allocate @param req Resource request]]> check to return true for each 1000 ms. See also {@link #waitFor(com.google.common.base.Supplier, int)} and {@link #waitFor(com.google.common.base.Supplier, int, int)} @param check]]> check to return true for each checkEveryMillis ms. See also {@link #waitFor(com.google.common.base.Supplier, int, int)} @param check user defined checker @param checkEveryMillis interval to call check]]> check to return true for each checkEveryMillis ms. In the main loop, this method will log the message "waiting in main loop" for each logInterval times iteration to confirm the thread is alive. @param check user defined checker @param checkEveryMillis interval to call check @param logInterval interval to log for each]]> AMRMClientAsync handles communication with the ResourceManager and provides asynchronous updates on events such as container allocations and completions. It contains a thread that sends periodic heartbeats to the ResourceManager. It should be used by implementing a CallbackHandler: 
     {@code
 class MyCallbackHandler implements AMRMClientAsync.CallbackHandler {
   public void onContainersAllocated(List containers) {
     [run tasks on the containers]
   }

   public void onContainersCompleted(List statuses) {
     [update progress, check whether app is done]
   }

   public void onNodesUpdated(List updated) {}

   public void onReboot() {}
 }
 }
    The client's lifecycle should be managed similarly to the following: 
     {@code
 AMRMClientAsync asyncClient =
     createAMRMClientAsync(appAttId, 1000, new MyCallbackhandler());
 asyncClient.init(conf);
 asyncClient.start();
 RegisterApplicationMasterResponse response = asyncClient
    .registerApplicationMaster(appMasterHostname, appMasterRpcPort,
       appMasterTrackingUrl);
 asyncClient.addContainerRequest(containerRequest);
 [... wait for application to complete]
 asyncClient.unregisterApplicationMaster(status, appMsg, trackingUrl);
 asyncClient.stop();
 }
    ]]>     NMClientAsync handles communication with all the NodeManagers and provides asynchronous updates on getting responses from them. It maintains a thread pool to communicate with individual NMs where a number of worker threads process requests to NMs by using {@link NMClientImpl}. The max size of the thread pool is configurable through {@link YarnConfiguration#NM_CLIENT_ASYNC_THREAD_POOL_MAX_SIZE}. It should be used in conjunction with a CallbackHandler. For example 
     {@code
 class MyCallbackHandler implements NMClientAsync.CallbackHandler {
   public void onContainerStarted(ContainerId containerId,
       Map allServiceResponse) {
     [post process after the container is started, process the response]
   }

   public void onContainerStatusReceived(ContainerId containerId,
       ContainerStatus containerStatus) {
     [make use of the status of the container]
   }

   public void onContainerStopped(ContainerId containerId) {
     [post process after the container is stopped]
   }

   public void onStartContainerError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }

   public void onGetContainerStatusError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }

   public void onStopContainerError(
       ContainerId containerId, Throwable t) {
     [handle the raised exception]
   }
 }
 }
    The client's life-cycle should be managed like the following: 
     {@code
 NMClientAsync asyncClient =
     NMClientAsync.createNMClientAsync(new MyCallbackhandler());
 asyncClient.init(conf);
 asyncClient.start();
 asyncClient.startContainer(container, containerLaunchContext);
 [... wait for container being started]
 asyncClient.getContainerStatus(container.getId(), container.getNodeId(),
     container.getContainerToken());
 [... handle the status in the callback instance]
 asyncClient.stopContainer(container.getId(), container.getNodeId(),
     container.getContainerToken());
 [... wait for container being stopped]
 asyncClient.stop();
 }
    ]]>     NodeManager responds to indicate its acceptance of the starting container request @param containerId the Id of the container @param allServiceResponse a Map between the auxiliary service names and their outputs]]> NodeManager responds with the status of the container @param containerId the Id of the container @param containerStatus the status of the container]]> NodeManager responds to indicate the container is stopped. @param containerId the Id of the container]]> The callback interface needs to be implemented by {@link NMClientAsync} users. The APIs are called when responses from NodeManager are available.

    Once a callback happens, the users can chose to act on it in blocking or non-blocking manner. If the action on callback is done in a blocking manner, some of the threads performing requests on NodeManagers may get blocked depending on how many threads in the pool are busy.

    The implementation of the callback function should not throw the unexpected exception. Otherwise, {@link NMClientAsync} will just catch, log and then ignore it.

    ]]>