YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan.
This commit is contained in:
Sunil G
parent
121e1e1280
commit
86b227a1fb
@ -0,0 +1,230 @@
|
||||
<!---
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License. See accompanying LICENSE file.
|
||||
-->
|
||||
|
||||
|
||||
# Using GPU On YARN
|
||||
# Prerequisites
|
||||
|
||||
- As of now, only Nvidia GPUs are supported by YARN
|
||||
- YARN node managers have to be pre-installed with Nvidia drivers.
|
||||
- When Docker is used as container runtime context, nvidia-docker 1.0 needs to be installed (Current supported version in YARN for nvidia-docker).
|
||||
|
||||
# Configs
|
||||
|
||||
## GPU scheduling
|
||||
|
||||
In `resource-types.xml`
|
||||
|
||||
Add following properties
|
||||
|
||||
```
|
||||
<configuration>
|
||||
<property>
|
||||
<name>yarn.resource-types</name>
|
||||
<value>yarn.io/gpu</value>
|
||||
</property>
|
||||
</configuration>
|
||||
```
|
||||
|
||||
In `yarn-site.xml`
|
||||
|
||||
`DominantResourceCalculator` MUST be configured to enable GPU scheduling/isolation.
|
||||
|
||||
For `Capacity Scheduler`, use following property to configure `DominantResourceCalculator` (In `capacity-scheduler.xml`):
|
||||
|
||||
| Property | Default value |
|
||||
| --- | --- |
|
||||
| yarn.scheduler.capacity.resource-calculator | org.apache.hadoop.yarn.util.resource.DominantResourceCalculator |
|
||||
|
||||
|
||||
## GPU Isolation
|
||||
|
||||
### In `yarn-site.xml`
|
||||
|
||||
```
|
||||
<property>
|
||||
<name>yarn.nodemanager.resource-plugins</name>
|
||||
<value>yarn.io/gpu</value>
|
||||
</property>
|
||||
```
|
||||
|
||||
This is to enable GPU isolation module on NodeManager side.
|
||||
|
||||
By default, YARN will automatically detect and config GPUs when above config is set. Following configs need to be set in `yarn-site.xml` only if admin has specialized requirements.
|
||||
|
||||
**1) Allowed GPU Devices**
|
||||
|
||||
| Property | Default value |
|
||||
| --- | --- |
|
||||
| yarn.nodemanager.resource-plugins.gpu.allowed-gpu-devices | auto |
|
||||
|
||||
Specify GPU devices which can be managed by YARN NodeManager (split by comma).
|
||||
Number of GPU devices will be reported to RM to make scheduling decisions.
|
||||
Set to auto (default) let YARN automatically discover GPU resource from
|
||||
system.
|
||||
|
||||
Manually specify GPU devices if auto detect GPU device failed or admin
|
||||
only want subset of GPU devices managed by YARN. GPU device is identified
|
||||
by their minor device number and index. A common approach to get minor
|
||||
device number of GPUs is using `nvidia-smi -q` and search `Minor Number`
|
||||
output.
|
||||
|
||||
When minor numbers are specified manually, admin needs to include indice of GPUs
|
||||
as well, format is `index:minor_number[,index:minor_number...]`. An example
|
||||
of manual specification is `0:0,1:1,2:2,3:4"`to allow YARN NodeManager to
|
||||
manage GPU devices with indices `0/1/2/3` and minor number `0/1/2/4`.
|
||||
numbers .
|
||||
|
||||
**2) Executable to discover GPUs**
|
||||
|
||||
| Property | value |
|
||||
| --- | --- |
|
||||
| yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables | /absolute/path/to/nvidia-smi |
|
||||
|
||||
When `yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto` specified,
|
||||
YARN NodeManager needs to run GPU discovery binary (now only support
|
||||
`nvidia-smi`) to get GPU-related information.
|
||||
When value is empty (default), YARN NodeManager will try to locate
|
||||
discovery executable itself.
|
||||
An example of the config value is: `/usr/local/bin/nvidia-smi`
|
||||
|
||||
**3) Docker Plugin Related Configs**
|
||||
|
||||
Following configs can be customized when user needs to run GPU applications inside Docker container. They're not required if admin follows default installation/configuration of `nvidia-docker`.
|
||||
|
||||
| Property | Default value |
|
||||
| --- | --- |
|
||||
| yarn.nodemanager.resource-plugins.gpu.docker-plugin | nvidia-docker-v1 |
|
||||
|
||||
Specify docker command plugin for GPU. By default uses Nvidia docker V1.0.
|
||||
|
||||
| Property | Default value |
|
||||
| --- | --- |
|
||||
| yarn.nodemanager.resource-plugins.gpu.docker-plugin.nvidia-docker-v1.endpoint | http://localhost:3476/v1.0/docker/cli |
|
||||
|
||||
Specify end point of `nvidia-docker-plugin`. Please find documentation: https://github.com/NVIDIA/nvidia-docker/wiki For more details.
|
||||
|
||||
**4) CGroups mount**
|
||||
|
||||
GPU isolation uses CGroup [devices controller](https://www.kernel.org/doc/Documentation/cgroup-v1/devices.txt) to do per-GPU device isolation. Following configs should be added to `yarn-site.xml` to automatically mount CGroup sub devices, otherwise admin has to manually create devices subfolder in order to use this feature.
|
||||
|
||||
| Property | Default value |
|
||||
| --- | --- |
|
||||
| yarn.nodemanager.linux-container-executor.cgroups.mount | true |
|
||||
|
||||
|
||||
### In `container-executor.cfg`
|
||||
|
||||
In general, following config needs to be added to `container-executor.cfg`
|
||||
|
||||
```
|
||||
[gpu]
|
||||
module.enabled=true
|
||||
```
|
||||
|
||||
When user needs to run GPU applications under non-Docker environment:
|
||||
|
||||
```
|
||||
[cgroups]
|
||||
# This should be same as yarn.nodemanager.linux-container-executor.cgroups.mount-path inside yarn-site.xml
|
||||
root=/sys/fs/cgroup
|
||||
# This should be same as yarn.nodemanager.linux-container-executor.cgroups.hierarchy inside yarn-site.xml
|
||||
yarn-hierarchy=yarn
|
||||
```
|
||||
|
||||
When user needs to run GPU applications under Docker environment:
|
||||
|
||||
**1) Add GPU related devices to docker section:**
|
||||
|
||||
Values separated by comma, you can get this by running `ls /dev/nvidia*`
|
||||
|
||||
```
|
||||
[docker]
|
||||
docker.allowed.devices=/dev/nvidiactl,/dev/nvidia-uvm,/dev/nvidia-uvm-tools,/dev/nvidia1,/dev/nvidia0
|
||||
```
|
||||
|
||||
**2) Add `nvidia-docker` to volume-driver whitelist.**
|
||||
|
||||
```
|
||||
[docker]
|
||||
...
|
||||
docker.allowed.volume-drivers
|
||||
```
|
||||
|
||||
**3) Add `nvidia_driver_<version>` to readonly mounts whitelist.**
|
||||
|
||||
```
|
||||
[docker]
|
||||
...
|
||||
docker.allowed.ro-mounts=nvidia_driver_375.66
|
||||
```
|
||||
|
||||
# Use it
|
||||
|
||||
## Distributed-shell + GPU
|
||||
|
||||
Distributed shell currently support specify additional resource types other than memory and vcores.
|
||||
|
||||
### Distributed-shell + GPU without Docker
|
||||
|
||||
Run distributed shell without using docker container (Asks 2 tasks, each task has 3GB memory, 1 vcore, 2 GPU device resource):
|
||||
|
||||
```
|
||||
yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
|
||||
-jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
|
||||
-shell_command /usr/local/nvidia/bin/nvidia-smi \
|
||||
-container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
|
||||
-num_containers 2
|
||||
```
|
||||
|
||||
You should be able to see output like
|
||||
|
||||
```
|
||||
Tue Dec 5 22:21:47 2017
|
||||
+-----------------------------------------------------------------------------+
|
||||
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|
||||
|-------------------------------+----------------------+----------------------+
|
||||
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
||||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|
||||
|===============================+======================+======================|
|
||||
| 0 Tesla P100-PCIE... Off | 0000:04:00.0 Off | 0 |
|
||||
| N/A 30C P0 24W / 250W | 0MiB / 12193MiB | 0% Default |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 1 Tesla P100-PCIE... Off | 0000:82:00.0 Off | 0 |
|
||||
| N/A 34C P0 25W / 250W | 0MiB / 12193MiB | 0% Default |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
|
||||
+-----------------------------------------------------------------------------+
|
||||
| Processes: GPU Memory |
|
||||
| GPU PID Type Process name Usage |
|
||||
|=============================================================================|
|
||||
| No running processes found |
|
||||
+-----------------------------------------------------------------------------+
|
||||
```
|
||||
|
||||
For launched container task.
|
||||
|
||||
### Distributed-shell + GPU with Docker
|
||||
|
||||
You can also run distributed shell with Docker container. `YARN_CONTAINER_RUNTIME_TYPE`/`YARN_CONTAINER_RUNTIME_DOCKER_IMAGE` must be specified to use docker container.
|
||||
|
||||
```
|
||||
yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
|
||||
-jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
|
||||
-shell_env YARN_CONTAINER_RUNTIME_TYPE=docker \
|
||||
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=<docker-image-name> \
|
||||
-shell_command nvidia-smi \
|
||||
-container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
|
||||
-num_containers 2
|
||||
```
|
||||
Loading…
Reference in New Issue
Block a user