big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

YARN-9326. Fair Scheduler configuration defaults are not documented in case of min and maxResources. Contributed by Adam Antal

Browse Source
This commit is contained in:
Szilard Nemeth 2019-07-15 13:28:01 +02:00
parent 18ee1092b4
commit 5446308360
1 changed files with 10 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
View File

@ -86,13 +86,19 @@ The allocation file must be in XML format. The format contains five types of ele
* **Queue elements**: which represent queues. Queue elements can take an optional attribute 'type', which when set to 'parent' makes it a parent queue. This is useful when we want to create a parent queue without configuring any leaf queues. Each queue element may contain the following properties: * **Queue elements**: which represent queues. Queue elements can take an optional attribute 'type', which when set to 'parent' makes it a parent queue. This is useful when we want to create a parent queue without configuring any leaf queues. Each queue element may contain the following properties:
* **minResources**: minimum resources the queue is entitled to, in the form of "X mb, Y vcores" or "vcores=X, memory-mb=Y". The latter form is required when specifying resources other than memory and CPU. For the single-resource fairness policy, the vcores value is ignored. If a queue's minimum share is not satisfied, it will be offered available resources before any other queue under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied if its memory usage is below its minimum memory share. Under dominant resource fairness, a queue is considered unsatisfied if its usage for its dominant resource with respect to the cluster capacity is below its minimum share for that resource. If multiple queues are unsatisfied in this situation, resources go to the queue with the smallest ratio between relevant resource usage and its minimum. Note that it is possible for a queue that is below its minimum to not immediately get up to its minimum when an application is submitted to the queue, because already-running jobs may be using those resources. * **minResources**: minimum resources the queue is entitled to. For the single-resource fairness policy, only the memory is used, other resources are ignored. If a queue's minimum share is not satisfied, it will be offered available resources before any other queue under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied if its memory usage is below its minimum memory share. Under dominant resource fairness, a queue is considered unsatisfied if its usage for its dominant resource with respect to the cluster capacity is below its minimum share for that resource. If multiple queues are unsatisfied in this situation, resources go to the queue with the smallest ratio between relevant resource usage and its minimum. Note that it is possible for a queue that is below its minimum to not immediately get up to its minimum when an application is submitted to the queue, because already-running jobs may be using those resources.
* **maxResources**: maximum resources a queue will allocated, expressed in the form of "X%", "X% cpu, Y% memory", "X mb, Y vcores", or "vcores=X, memory-mb=Y". The last form is required when specifying resources other than memory and CPU. In the last form, X and Y can either be a percentage or an integer resource value without units. In the latter case the units will be inferred from the default units configured for that resource. A queue will not be assigned a container that would put its aggregate usage over this limit. * **maxResources**: maximum resources a queue can be allocated. A queue will not be assigned a container that would put its aggregate usage over this limit. This limit is enforced recursively, the queue will not be assigned a container if that assignment would put the queue or its parent(s) over the maximum resources.
* **maxContainerAllocation**: maximum resources a queue can allocate for a single container, expressed in the form of "X mb, Y vcores" or "vcores=X, memory-mb=Y". The latter form is required when specifying resources other than memory and CPU. If the property is not set it's value is inherited from a parent queue. It's default value is **yarn.scheduler.maximum-allocation-mb**. Cannot be higher than **maxResources**. This property is invalid for root queue. * **maxContainerAllocation**: maximum resources a queue can allocate for a single container. If the property is not set it's value is inherited from a parent queue. The default values are **yarn.scheduler.maximum-allocation-mb** and **yarn.scheduler.maximum-allocation-vcores**. Cannot be higher than **maxResources**. This property is invalid for root queue.
* **maxChildResources**: maximum resources an ad hoc child queue will allocated, expressed in the form of "X%", "X% cpu, Y% memory", "X mb, Y vcores", or "vcores=X, memory-mb=Y". The last form is required when specifying resources other than memory and CPU. In the last form, X and Y can either be a percentage or an integer resource value without units. In the latter case the units will be inferred from the default units configured for that resource. An ad hoc child queue will not be assigned a container that would put its aggregate usage over this limit. * **maxChildResources**: maximum resources an ad hoc child queue can be allocated. A child queue limit is enforced recursively and thus will not be assigned a container if that assignment would put the child queue or its parent(s) over the maximum resources.
* For minResources, maxResources, maxContainerAllocation and maxChildResources properties one can give the parameter in one of the following formats:
* *Old format*: "X mb, Y vcores", "X% cpu, Y% memory", "X%". When not a single percentage is provided, configuring both memory and cpu is mandatory, while other resource types are ignored, and set to zero.
* *New format (recommended)*: "vcores=X, memory-mb=Y" or "vcores=X%, memory-mb=Y%". As seen, in this format one can give either a percentage or an integer resource value without units. In the latter case the units will be inferred from the default units configured for that resource. This format is required when resources other than memory and CPU are specified. Any unspecified resource will be set to 0 in case of minResources, or to the maximum for that resource in case of maxResources, maxContainerAllocation and maxChildResources.
* **maxRunningApps**: limit the number of apps from the queue to run at once * **maxRunningApps**: limit the number of apps from the queue to run at once