hadoop/hadoop-dist/src/main/compose

Docker cluster definitions

This directory contains multiple docker cluster definitions to start local pseudo cluster with different configuration.

It helps to start local (multi-node like) pseudo cluster with docker and docker-compose and obviously it's not for production.

You may find more information in the specific subdirectories but in generic you can use the following commands:

Usage

To start a cluster go to a subdirectory and start the cluster:

docker-compose up -d

You can check the logs of all the components with:

docker-compose logs

In case of a problem you can destroy the cluster an delete all the local state with:

docker-compose down

(Note: a simple docker-compose stop may not delete all the local data).

You can scale up and down the components:

docker-compose scale datanode=5

Usually the key webui ports are published on the docker host.