big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDDS-2002. Update documentation for 0.4.1 release.

Browse Source 
Signed-off-by: Anu Engineer <aengineer@apache.org>
This commit is contained in:
Nanda kumar 2019-08-21 22:47:41 +05:30 committed by Anu Engineer
parent 0b796754b9
commit b661dcf563
25 changed files with 269 additions and 258 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
hadoop-hdds/docs/content/beyond/Containers.md
View File

@ -25,8 +25,9 @@ Docker heavily is used at the ozone development with three principal use-cases:
* __dev__: * __dev__:
* We use docker to start local pseudo-clusters (docker provides unified environment, but no image creation is required) * We use docker to start local pseudo-clusters (docker provides unified environment, but no image creation is required)
* __test__: * __test__:
* We create docker images from the dev branches to test ozone in kubernetes and other container orchestator system * We create docker images from the dev branches to test ozone in kubernetes and other container orchestrator system
* We provide _apache/ozone_ images for each release to make it easier the evaluation of Ozone. These images are __not__ created __for production__ usage. * We provide _apache/ozone_ images for each release to make it easier for evaluation of Ozone.
These images are __not__ created __for production__ usage.
<div class="alert alert-warning" role="alert"> <div class="alert alert-warning" role="alert">
We <b>strongly</b> recommend that you create your own custom images when you We <b>strongly</b> recommend that you create your own custom images when you
@ -36,7 +37,7 @@ shipped container images and k8s resources as examples and guides to help you
</div> </div>
* __production__: * __production__:
* We document how can you create your own docker image for your production cluster. * We have documentation on how you can create your own docker image for your production cluster.
Let's check out each of the use-cases in more detail: Let's check out each of the use-cases in more detail:
@ -46,38 +47,41 @@ Ozone artifact contains example docker-compose directories to make it easier to
From distribution: From distribution:
``` ```bash
cd compose/ozone cd compose/ozone
docker-compose up -d docker-compose up -d
``` ```
After a local build After a local build:
``` ```bash
cd hadoop-ozone/dist/target/ozone-*/compose cd hadoop-ozone/dist/target/ozone-*/compose
docker-compose up -d docker-compose up -d
``` ```
These environments are very important tools to start different type of Ozone clusters at any time. These environments are very important tools to start different type of Ozone clusters at any time.
To be sure that the compose files are up-to-date, we also provide acceptance test suites which start the cluster and check the basic behaviour. To be sure that the compose files are up-to-date, we also provide acceptance test suites which start
the cluster and check the basic behaviour.
The acceptance tests are part of the distribution, and you can find the test definitions in `./smoketest` directory. The acceptance tests are part of the distribution, and you can find the test definitions in `smoketest` directory.
You can start the tests from any compose directory: You can start the tests from any compose directory:
For example: For example:
``` ```bash
cd compose/ozone cd compose/ozone
./test.sh ./test.sh
``` ```
### Implementation details ### Implementation details
`./compose` tests are based on the apache/hadoop-runner docker image. The image itself doesn't contain any Ozone jar file or binary just the helper scripts to start ozone. `compose` tests are based on the apache/hadoop-runner docker image. The image itself does not contain
any Ozone jar file or binary just the helper scripts to start ozone.
hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone distribution itself is mounted from the including directory: hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone distribution itself
is mounted from the including directory:
(Example docker-compose fragment) (Example docker-compose fragment)
@ -91,7 +95,9 @@ hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone
``` ```
The containers are conigured based on environment variables, but because the same environment variables should be set for each containers we maintain the list of the environment variables in a separated file: The containers are configured based on environment variables, but because the same environment
variables should be set for each containers we maintain the list of the environment variables
in a separated file:
``` ```
scm: scm:
@ -111,23 +117,32 @@ OZONE-SITE.XML_ozone.enabled=True
#... #...
``` ```
As you can see we use naming convention. Based on the name of the environment variable, the appropariate hadoop config XML (`ozone-site.xml` in our case) will be generated by a [script](https://github.com/apache/hadoop/tree/docker-hadoop-runner-latest/scripts) which is included in the `hadoop-runner` base image. As you can see we use naming convention. Based on the name of the environment variable, the
appropriate hadoop config XML (`ozone-site.xml` in our case) will be generated by a
[script](https://github.com/apache/hadoop/tree/docker-hadoop-runner-latest/scripts) which is
included in the `hadoop-runner` base image.
The [entrypoint](https://github.com/apache/hadoop/blob/docker-hadoop-runner-latest/scripts/starter.sh) of the `hadoop-runner` image contains a helper shell script which triggers this transformation and cab do additional actions (eg. initialize scm/om storage, download required keytabs, etc.) based on environment variables. The [entrypoint](https://github.com/apache/hadoop/blob/docker-hadoop-runner-latest/scripts/starter.sh)
of the `hadoop-runner` image contains a helper shell script which triggers this transformation and
can do additional actions (eg. initialize scm/om storage, download required keytabs, etc.)
based on environment variables.
## Test/Staging ## Test/Staging
The `docker-compose` based approach is recommended only for local test not for multi node cluster. To use containers on a multi-node cluster we need a Container Orchestrator like Kubernetes. The `docker-compose` based approach is recommended only for local test, not for multi node cluster.
To use containers on a multi-node cluster we need a Container Orchestrator like Kubernetes.
Kubernetes example files are included in the `kubernetes` folder. Kubernetes example files are included in the `kubernetes` folder.
*Please note*: all the provided images are based the `hadoop-runner` image which contains all the required tool for testing in staging environments. For production we recommend to create your own, hardened image with your own base image. *Please note*: all the provided images are based the `hadoop-runner` image which contains all the
required tool for testing in staging environments. For production we recommend to create your own,
hardened image with your own base image.
### Test the release ### Test the release
The release can be tested with deploying any of the example clusters: The release can be tested with deploying any of the example clusters:
``` ```bash
cd kubernetes/examples/ozone cd kubernetes/examples/ozone
kubectl apply -f kubectl apply -f
``` ```
@ -139,13 +154,13 @@ Plese note that in this case the latest released container will be downloaded fr
To test a development build you can create your own image and upload it to your own docker registry: To test a development build you can create your own image and upload it to your own docker registry:
``` ```bash
mvn clean install -f pom.ozone.xml -DskipTests -Pdocker-build,docker-push -Ddocker.image=myregistry:9000/name/ozone mvn clean install -f pom.ozone.xml -DskipTests -Pdocker-build,docker-push -Ddocker.image=myregistry:9000/name/ozone
``` ```
The configured image will be used in all the generated kubernetes resources files (`image:` keys are adjusted during the build) The configured image will be used in all the generated kubernetes resources files (`image:` keys are adjusted during the build)
``` ```bash
cd kubernetes/examples/ozone cd kubernetes/examples/ozone
kubectl apply -f kubectl apply -f
``` ```
@ -160,10 +175,12 @@ adjust base image, umask, security settings, user settings according to your own
You can use the source of our development images as an example: You can use the source of our development images as an example:
* Base image: https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile * [Base image] (https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile)
* Docker image: https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/src/main/Dockerfile * [Docker image] (https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/src/main/docker/Dockerfile)
Most of the elements are optional and just helper function but to use the provided example kubernetes resources you may need the scripts from [here](https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts) Most of the elements are optional and just helper function but to use the provided example
kubernetes resources you may need the scripts from
[here](https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts)
* The two python scripts convert environment variables to real hadoop XML config files * The two python scripts convert environment variables to real hadoop XML config files
* The start.sh executes the python scripts (and other initialization) based on environment variables. * The start.sh executes the python scripts (and other initialization) based on environment variables.
@ -205,7 +222,7 @@ Ozone related container images and source locations:
<td>This is the base image used for testing Hadoop Ozone. <td>This is the base image used for testing Hadoop Ozone.
This is a set of utilities that make it easy for us run ozone.</td> This is a set of utilities that make it easy for us run ozone.</td>
</tr> </tr>
<tr> <!---tr>
<th scope="row">3</th> <th scope="row">3</th>
<td>apache/ozone:build (WIP)</td> <td>apache/ozone:build (WIP)</td>
<td>https://github.com/apache/hadoop-docker-ozone</td> <td>https://github.com/apache/hadoop-docker-ozone</td>
@ -213,6 +230,6 @@ Ozone related container images and source locations:
<td> </td> <td> </td>
<td> </td> <td> </td>
<td>TODO: Add more documentation here.</td> <td>TODO: Add more documentation here.</td>
</tr> </tr-->
</tbody> </tbody>
</table> </table>

7
hadoop-hdds/docs/content/beyond/DockerCheatSheet.md
View File

@ -22,7 +22,9 @@ weight: 4
limitations under the License. limitations under the License.
--> -->
In the `compose` directory of the ozone distribution there are multiple pseudo-cluster setup which can be used to run Ozone in different way (for example with secure cluster, with tracing enabled, with prometheus etc.). In the `compose` directory of the ozone distribution there are multiple pseudo-cluster setup which
can be used to run Ozone in different way (for example: secure cluster, with tracing enabled,
with prometheus etc.).
If the usage is not document in a specific directory the default usage is the following: If the usage is not document in a specific directory the default usage is the following:
@ -31,8 +33,7 @@ cd compose/ozone
docker-compose up -d docker-compose up -d
``` ```
The data of the container is ephemeral and deleted together with the docker volumes. To force the deletion of existing data you can always delete all the temporary data: The data of the container is ephemeral and deleted together with the docker volumes.
```bash ```bash
docker-compose down docker-compose down
``` ```

2
hadoop-hdds/docs/content/beyond/RunningWithHDFS.md
View File

@ -56,7 +56,7 @@ To start ozone with HDFS you should start the the following components:
2. HDFS Datanode (from the Hadoop distribution with the plugin on the 2. HDFS Datanode (from the Hadoop distribution with the plugin on the
classpath from the Ozone distribution) classpath from the Ozone distribution)
3. Ozone Manager (from the Ozone distribution) 3. Ozone Manager (from the Ozone distribution)
4. Storage Container manager (from the Ozone distribution) 4. Storage Container Manager (from the Ozone distribution)
Please check the log of the datanode whether the HDDS/Ozone plugin is started or Please check the log of the datanode whether the HDDS/Ozone plugin is started or
not. Log of datanode should contain something like this: not. Log of datanode should contain something like this:

6
hadoop-hdds/docs/content/concept/Datanodes.md
View File

@ -36,7 +36,7 @@ actual data streams. This is the default Storage container format. From
Ozone's perspective, container is a protocol spec, actual storage layouts Ozone's perspective, container is a protocol spec, actual storage layouts
does not matter. In other words, it is trivial to extend or bring new does not matter. In other words, it is trivial to extend or bring new
container layouts. Hence this should be treated as a reference implementation container layouts. Hence this should be treated as a reference implementation
of containers under Ozone. of containers under Ozone.
## Understanding Ozone Blocks and Containers ## Understanding Ozone Blocks and Containers
@ -51,13 +51,13 @@ shows the logical layout out of Ozone block.
The container ID lets the clients discover the location of the container. The The container ID lets the clients discover the location of the container. The
authoritative information about where a container is located is with the authoritative information about where a container is located is with the
Storage Container Manager or SCM. In most cases, the container location will Storage Container Manager (SCM). In most cases, the container location will be
cached by Ozone Manager and will be returned along with the Ozone blocks. cached by Ozone Manager and will be returned along with the Ozone blocks.
Once the client is able to locate the contianer, that is, understand which Once the client is able to locate the contianer, that is, understand which
data nodes contain this container, the client will connect to the datanode data nodes contain this container, the client will connect to the datanode
read the data the data stream specified by container ID:Local ID. In other and read the data stream specified by _Container ID:Local ID_. In other
words, the local ID serves as index into the container which describes what words, the local ID serves as index into the container which describes what
data stream we want to read from. data stream we want to read from.

2
hadoop-hdds/docs/content/concept/Hdds.md
View File

@ -23,7 +23,7 @@ summary: Storage Container Manager or SCM is the core metadata service of Ozone
Storage container manager provides multiple critical functions for the Ozone Storage container manager provides multiple critical functions for the Ozone
cluster. SCM acts as the cluster manager, Certificate authority, Block cluster. SCM acts as the cluster manager, Certificate authority, Block
manager and the replica manager. manager and the Replica manager.
{{<card title="Cluster Management" icon="tasks">}} {{<card title="Cluster Management" icon="tasks">}}
SCM is in charge of creating an Ozone cluster. When an SCM is booted up via <kbd>init</kbd> command, SCM creates the cluster identity and root certificates needed for the SCM certificate authority. SCM manages the life cycle of a data node in the cluster. SCM is in charge of creating an Ozone cluster. When an SCM is booted up via <kbd>init</kbd> command, SCM creates the cluster identity and root certificates needed for the SCM certificate authority. SCM manages the life cycle of a data node in the cluster.

6
hadoop-hdds/docs/content/concept/Overview.md
View File

@ -56,7 +56,7 @@ Ozone.
![FunctionalOzone](FunctionalOzone.png) ![FunctionalOzone](FunctionalOzone.png)
Any distributed system can viewed from different perspectives. One way to Any distributed system can be viewed from different perspectives. One way to
look at Ozone is to imagine it as Ozone Manager as a name space service built on look at Ozone is to imagine it as Ozone Manager as a name space service built on
top of HDDS, a distributed block store. top of HDDS, a distributed block store.
@ -67,8 +67,8 @@ Another way to visualize Ozone is to look at the functional layers; we have a
We have a data storage layer, which is basically the data nodes and they are We have a data storage layer, which is basically the data nodes and they are
managed by SCM. managed by SCM.
The replication layer, provided by Ratis is used to replicate metadata (Ozone The replication layer, provided by Ratis is used to replicate metadata (OM and SCM)
Manager and SCM) and also used for consistency when data is modified at the and also used for consistency when data is modified at the
data nodes. data nodes.
We have a management server called Recon, that talks to all other components We have a management server called Recon, that talks to all other components

20
hadoop-hdds/docs/content/concept/OzoneManager.md
View File

@ -21,14 +21,14 @@ summary: Ozone Manager is the principal name space service of Ozone. OM manages
limitations under the License. limitations under the License.
--> -->
Ozone Manager or OM is the namespace manager for Ozone. Ozone Manager (OM) is the namespace manager for Ozone.
This means that when you want to write some data, you ask Ozone This means that when you want to write some data, you ask Ozone
manager for a block and Ozone Manager gives you a block and remembers that Manager for a block and Ozone Manager gives you a block and remembers that
information. When you want to read the that file back, you need to find the information. When you want to read that file back, you need to find the
address of the block and Ozone manager returns it you. address of the block and Ozone Manager returns it you.
Ozone manager also allows users to organize keys under a volume and bucket. Ozone Manager also allows users to organize keys under a volume and bucket.
Volumes and buckets are part of the namespace and managed by Ozone Manager. Volumes and buckets are part of the namespace and managed by Ozone Manager.
Each ozone volume is the root of an independent namespace under OM. Each ozone volume is the root of an independent namespace under OM.
@ -57,17 +57,17 @@ understood if we trace what happens during a key write and key read.
* To write a key to Ozone, a client tells Ozone manager that it would like to * To write a key to Ozone, a client tells Ozone manager that it would like to
write a key into a bucket that lives inside a specific volume. Once Ozone write a key into a bucket that lives inside a specific volume. Once Ozone
manager determines that you are allowed to write a key to specified bucket, Manager determines that you are allowed to write a key to the specified bucket,
OM needs to allocate a block for the client to write data. OM needs to allocate a block for the client to write data.
* To allocate a block, Ozone manager sends a request to Storage Container * To allocate a block, Ozone Manager sends a request to Storage Container
Manager or SCM; SCM is the manager of data nodes. SCM picks three data nodes Manager (SCM); SCM is the manager of data nodes. SCM picks three data nodes
into which client can write data. SCM allocates the block and returns the into which client can write data. SCM allocates the block and returns the
block ID to Ozone Manager. block ID to Ozone Manager.
* Ozone manager records this block information in its metadata and returns the * Ozone manager records this block information in its metadata and returns the
block and a block token (a security permission to write data to the block) block and a block token (a security permission to write data to the block)
the client. to the client.
* The client uses the block token to prove that it is allowed to write data to * The client uses the block token to prove that it is allowed to write data to
the block and writes data to the data node. the block and writes data to the data node.
@ -82,6 +82,6 @@ Ozone manager.
* Key reads are simpler, the client requests the block list from the Ozone * Key reads are simpler, the client requests the block list from the Ozone
Manager Manager
* Ozone manager will return the block list and block tokens which * Ozone manager will return the block list and block tokens which
allows the client to read the data from nodes. allows the client to read the data from data nodes.
* Client connects to the data node and presents the block token and reads * Client connects to the data node and presents the block token and reads
the data from the data node. the data from the data node.

8
hadoop-hdds/docs/content/interface/JavaApi.md
View File

@ -74,21 +74,21 @@ It is possible to pass an array of arguments to the createVolume by creating vol
Once you have a volume, you can create buckets inside the volume. Once you have a volume, you can create buckets inside the volume.
{{< highlight bash >}} {{< highlight java >}}
// Let us create a bucket called videos. // Let us create a bucket called videos.
assets.createBucket("videos"); assets.createBucket("videos");
OzoneBucket video = assets.getBucket("videos"); OzoneBucket video = assets.getBucket("videos");
{{< /highlight >}} {{< /highlight >}}
At this point we have a usable volume and a bucket. Our volume is called assets and bucket is called videos. At this point we have a usable volume and a bucket. Our volume is called _assets_ and bucket is called _videos_.
Now we can create a Key. Now we can create a Key.
### Reading and Writing a Key ### Reading and Writing a Key
With a bucket object the users can now read and write keys. The following code reads a video called intro.mp4 from the local disk and stores in the video bucket that we just created. With a bucket object the users can now read and write keys. The following code reads a video called intro.mp4 from the local disk and stores in the _video_ bucket that we just created.
{{< highlight bash >}} {{< highlight java >}}
// read data from the file, this is a user provided function. // read data from the file, this is a user provided function.
byte [] videoData = readFile("intro.mp4"); byte [] videoData = readFile("intro.mp4");

8
hadoop-hdds/docs/content/interface/OzoneFS.md
View File

@ -21,7 +21,7 @@ summary: Hadoop Compatible file system allows any application that expects an HD
limitations under the License. limitations under the License.
--> -->
The Hadoop compatible file system interface allpws storage backends like Ozone The Hadoop compatible file system interface allows storage backends like Ozone
to be easily integrated into Hadoop eco-system. Ozone file system is an to be easily integrated into Hadoop eco-system. Ozone file system is an
Hadoop compatible file system. Hadoop compatible file system.
@ -36,7 +36,7 @@ ozone sh volume create /volume
ozone sh bucket create /volume/bucket ozone sh bucket create /volume/bucket
{{< /highlight >}} {{< /highlight >}}
Once this is created, please make sure that bucket exists via the listVolume or listBucket commands. Once this is created, please make sure that bucket exists via the _list volume_ or _list bucket_ commands.
Please add the following entry to the core-site.xml. Please add the following entry to the core-site.xml.
@ -45,6 +45,10 @@ Please add the following entry to the core-site.xml.
<name>fs.o3fs.impl</name> <name>fs.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value> <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
</property> </property>
<property>
<name>fs.AbstractFileSystem.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.OzFs</value>
</property>
<property> <property>
<name>fs.defaultFS</name> <name>fs.defaultFS</name>
<value>o3fs://bucket.volume</value> <value>o3fs://bucket.volume</value>

18
hadoop-hdds/docs/content/interface/S3.md
View File

@ -26,7 +26,7 @@ Ozone provides S3 compatible REST interface to use the object store data with an
## Getting started ## Getting started
S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components. S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components.
You can start a docker based cluster, including the S3 gateway from the release package. You can start a docker based cluster, including the S3 gateway from the release package.
@ -93,7 +93,7 @@ If security is not enabled, you can *use* **any** AWS_ACCESS_KEY_ID and AWS_SECR
If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*kerberos based authentication is required). If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*kerberos based authentication is required).
``` ```bash
/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
ozone s3 getsecret ozone s3 getsecret
awsAccessKey=testuser/scm@EXAMPLE.COM awsAccessKey=testuser/scm@EXAMPLE.COM
@ -103,7 +103,7 @@ awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
Now, you can use the key and the secret to access the S3 endpoint: Now, you can use the key and the secret to access the S3 endpoint:
``` ```bash
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999 export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1 aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1
@ -116,7 +116,7 @@ aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1
To show the storage location of a S3 bucket, use the `ozone s3 path <bucketname>` command. To show the storage location of a S3 bucket, use the `ozone s3 path <bucketname>` command.
``` ```bash
aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1 aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1
ozone s3 path bucket1 ozone s3 path bucket1
@ -128,23 +128,23 @@ Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
### AWS Cli ### AWS Cli
`aws` CLI could be used with specifying the custom REST endpoint. `aws` CLI could be used by specifying the custom REST endpoint.
``` ```bash
aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
``` ```
Or Or
``` ```bash
aws s3 ls --endpoint http://localhost:9878 s3://buckettest aws s3 ls --endpoint http://localhost:9878 s3://buckettest
``` ```
### S3 Fuse driver (goofys) ### S3 Fuse driver (goofys)
Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system: Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system.
``` ```bash
goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1 goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
``` ```

22
hadoop-hdds/docs/content/recipe/Prometheus.md
View File

@ -32,28 +32,29 @@ compatible metrics endpoint where all the available hadoop metrics are published
## Monitoring with prometheus ## Monitoring with prometheus
(1) To enable the Prometheus metrics endpoint you need to add a new configuration to the `ozone-site.xml` file: * To enable the Prometheus metrics endpoint you need to add a new configuration to the `ozone-site.xml` file.
``` ```xml
<property> <property>
<name>hdds.prometheus.endpoint.enabled</name> <name>hdds.prometheus.endpoint.enabled</name>
<value>true</value> <value>true</value>
</property> </property>
``` ```
_Note_: for Docker compose based pseudo cluster put the `OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true` line to the `docker-config` file. _Note_: for Docker compose based pseudo cluster put the \
`OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true` line to the `docker-config` file.
(2) Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints: * Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints:
* http://scm:9874/prom * http://scm:9874/prom
* http://ozoneManager:9876/prom * http://ozoneManager:9876/prom
(3) Create a prometheus.yaml configuration with the previous endpoints: * Create a prometheus.yaml configuration with the previous endpoints:
```yaml ```yaml
global: global:
scrape_interval: 15s scrape_interval: 15s
scrape_configs: scrape_configs:
- job_name: ozone - job_name: ozone
@ -64,20 +65,21 @@ scrape_configs:
- "ozoneManager:9874" - "ozoneManager:9874"
``` ```
(4) Start with prometheus from the directory where you have the prometheus.yaml file: * Start with prometheus from the directory where you have the prometheus.yaml file:
``` ```bash
prometheus prometheus
``` ```
(5) Check the active targets in the prometheus web-ui: * Check the active targets in the prometheus web-ui:
http://localhost:9090/targets http://localhost:9090/targets
![Prometheus target page example](prometheus.png) ![Prometheus target page example](prometheus.png)
(6) Check any metrics on the prometheus web ui. For example: * Check any metrics on the prometheus web ui.\
For example:
http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1 http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1

38
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md
View File

@ -46,13 +46,13 @@ You also need the following:
First of all create a docker image with the Spark image creator. First of all create a docker image with the Spark image creator.
Execute the following from the Spark distribution Execute the following from the Spark distribution
``` ```bash
./bin/docker-image-tool.sh -r myrepo -t 2.4.0 build ./bin/docker-image-tool.sh -r myrepo -t 2.4.0 build
``` ```
_Note_: if you use Minikube add the `-m` flag to use the docker daemon of the Minikube image: _Note_: if you use Minikube add the `-m` flag to use the docker daemon of the Minikube image:
``` ```bash
./bin/docker-image-tool.sh -m -r myrepo -t 2.4.0 build ./bin/docker-image-tool.sh -m -r myrepo -t 2.4.0 build
``` ```
@ -64,18 +64,22 @@ Create a new directory for customizing the created docker image.
Copy the `ozone-site.xml` from the cluster: Copy the `ozone-site.xml` from the cluster:
``` ```bash
kubectl cp om-0:/opt/hadoop/etc/hadoop/ozone-site.xml . kubectl cp om-0:/opt/hadoop/etc/hadoop/ozone-site.xml .
``` ```
And create a custom `core-site.xml`: And create a custom `core-site.xml`.
``` ```xml
<configuration> <configuration>
<property> <property>
<name>fs.o3fs.impl</name> <name>fs.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value> <value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value>
</property> </property>
<property>
<name>fs.AbstractFileSystem.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.OzFs</value>
</property>
</configuration> </configuration>
``` ```
@ -98,13 +102,13 @@ ENV SPARK_EXTRA_CLASSPATH=/opt/hadoop/conf
ADD hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar /opt/hadoop-ozone-filesystem-lib-legacy.jar ADD hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar /opt/hadoop-ozone-filesystem-lib-legacy.jar
``` ```
``` ```bash
docker build -t myrepo/spark-ozone docker build -t myrepo/spark-ozone
``` ```
For remote kubernetes cluster you may need to push it: For remote kubernetes cluster you may need to push it:
``` ```bash
docker push myrepo/spark-ozone docker push myrepo/spark-ozone
``` ```
@ -112,7 +116,7 @@ docker push myrepo/spark-ozone
Download any text file and put it to the `/tmp/alice.txt` first. Download any text file and put it to the `/tmp/alice.txt` first.
``` ```bash
kubectl port-forward s3g-0 9878:9878 kubectl port-forward s3g-0 9878:9878
aws s3api --endpoint http://localhost:9878 create-bucket --bucket=test aws s3api --endpoint http://localhost:9878 create-bucket --bucket=test
aws s3api --endpoint http://localhost:9878 put-object --bucket test --key alice.txt --body /tmp/alice.txt aws s3api --endpoint http://localhost:9878 put-object --bucket test --key alice.txt --body /tmp/alice.txt
@ -130,7 +134,7 @@ Write down the ozone filesystem uri as it should be used with the spark-submit c
## Create service account to use ## Create service account to use
``` ```bash
kubectl create serviceaccount spark -n yournamespace kubectl create serviceaccount spark -n yournamespace
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=yournamespace:spark --namespace=yournamespace kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=yournamespace:spark --namespace=yournamespace
``` ```
@ -138,13 +142,14 @@ kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount
Execute the following spark-submit command, but change at least the following values: Execute the following spark-submit command, but change at least the following values:
* the kubernetes master url (you can check your ~/.kube/config to find the actual value) * the kubernetes master url (you can check your _~/.kube/config_ to find the actual value)
* the kubernetes namespace (yournamespace in this example) * the kubernetes namespace (_yournamespace_ in this example)
* serviceAccountName (you can use the _spark_ value if you folllowed the previous steps) * serviceAccountName (you can use the _spark_ value if you followed the previous steps)
* container.image (in this example this is myrepo/spark-ozone. This is pushed to the registry in the previous steps) * container.image (in this example this is _myrepo/spark-ozone_. This is pushed to the registry in the previous steps)
* location of the input file (o3fs://...), use the string which is identified earlier with the `ozone s3 path <bucketname>` command * location of the input file (o3fs://...), use the string which is identified earlier with the \
`ozone s3 path <bucketname>` command
``` ```bash
bin/spark-submit \ bin/spark-submit \
--master k8s://https://kubernetes:6443 \ --master k8s://https://kubernetes:6443 \
--deploy-mode cluster \ --deploy-mode cluster \
@ -162,7 +167,8 @@ bin/spark-submit \
Check the available `spark-word-count-...` pods with `kubectl get pod` Check the available `spark-word-count-...` pods with `kubectl get pod`
Check the output of the calculation with `kubectl logs spark-word-count-1549973913699-driver` Check the output of the calculation with \
`kubectl logs spark-word-count-1549973913699-driver`
You should see the output of the wordcount job. For example: You should see the output of the wordcount job. For example:

3
hadoop-hdds/docs/content/recipe/_index.md
View File

@ -24,5 +24,6 @@ weight: 8
{{<jumbotron title="Recipes of Ozone">}} {{<jumbotron title="Recipes of Ozone">}}
Standard How-to documents which describe how to use Ozone with other Software. For example, How to use Ozone with Apache Spark. Standard how-to documents which describe how to use Ozone with other Software.
For example, how to use Ozone with Apache Spark.
{{</jumbotron>}} {{</jumbotron>}}

4
hadoop-hdds/docs/content/security/SecuityWithRanger.md
View File

@ -24,8 +24,8 @@ icon: user
Apache Ranger™ is a framework to enable, monitor and manage comprehensive data Apache Ranger™ is a framework to enable, monitor and manage comprehensive data
security across the Hadoop platform. The next version(any version after 1.20) security across the Hadoop platform. Any version of Apache Ranger which is greater
of Apache Ranger is aware of Ozone, and can manage an Ozone cluster. than 1.20 is aware of Ozone, and can manage an Ozone cluster.
To use Apache Ranger, you must have Apache Ranger installed in your Hadoop To use Apache Ranger, you must have Apache Ranger installed in your Hadoop

169
hadoop-hdds/docs/content/security/SecureOzone.md
View File

@ -31,11 +31,13 @@ secure networks where it is possible to deploy without securing the cluster.
This release of Ozone follows that model, but soon will move to _secure by This release of Ozone follows that model, but soon will move to _secure by
default._ Today to enable security in ozone cluster, we need to set the default._ Today to enable security in ozone cluster, we need to set the
configuration **ozone.security.enabled** to true. configuration **ozone.security.enabled** to _true_ and **hadoop.security.authentication**
to _kerberos_.
Property|Value Property|Value
----------------------|--------- ----------------------|---------
ozone.security.enabled| **true** ozone.security.enabled| _true_
hadoop.security.authentication| _kerberos_
# Tokens # # Tokens #
@ -68,7 +70,7 @@ also enabled by default when security is enabled.
Each of the service daemons that make up Ozone needs a Kerberos service Each of the service daemons that make up Ozone needs a Kerberos service
principal name and a corresponding [kerberos key tab]({{https://web.mit.edu/kerberos/krb5-latest/doc/basic/keytab_def.html}}) file. principal name and a corresponding [kerberos key tab](https://web.mit.edu/kerberos/krb5-latest/doc/basic/keytab_def.html) file.
All these settings should be made in ozone-site.xml. All these settings should be made in ozone-site.xml.
@ -77,101 +79,100 @@ All these settings should be made in ozone-site.xml.
<div class="card-body"> <div class="card-body">
<h3 class="card-title">Storage Container Manager</h3> <h3 class="card-title">Storage Container Manager</h3>
<p class="card-text"> <p class="card-text">
<br> <br>
SCM requires two Kerberos principals, and the corresponding key tab files SCM requires two Kerberos principals, and the corresponding key tab files
for both of these principals. for both of these principals.
<br> <br>
<table class="table table-dark"> <table class="table table-dark">
<thead> <thead>
<tr> <tr>
<th scope="col">Property</th> <th scope="col">Property</th>
<th scope="col">Description</th> <th scope="col">Description</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<th scope="row">hdds.scm.kerberos.principal</th> <td>hdds.scm.kerberos.principal</th>
<td>The SCM service principal. e.g. scm/HOST@REALM.COM</td> <td>The SCM service principal. <br/> e.g. scm/_HOST@REALM.COM</td>
</tr> </tr>
<tr> <tr>
<th scope="row">hdds.scm.kerberos.keytab.file</th> <td>hdds.scm.kerberos.keytab.file</th>
<td>The keytab file used by SCM daemon to login as its service principal.</td> <td>The keytab file used by SCM daemon to login as its service principal.</td>
</tr> </tr>
<tr> <tr>
<th scope="row">hdds.scm.http.kerberos.principal</th> <td>hdds.scm.http.kerberos.principal</th>
<td>SCM http server service principal.</td> <td>SCM http server service principal.</td>
</tr> </tr>
<tr> <tr>
<th scope="row">hdds.scm.http.kerberos.keytab</th> <td>hdds.scm.http.kerberos.keytab</th>
<td>The keytab file used by SCM http server to login as its service principal.</td> <td>The keytab file used by SCM http server to login as its service principal.</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
</div> </div>
</div> </div>
<div class="card"> <div class="card">
<div class="card-body"> <div class="card-body">
<h3 class="card-title">Ozone Manager</h3> <h3 class="card-title">Ozone Manager</h3>
<p class="card-text"> <p class="card-text">
<br> <br>
Like SCM, OM also requires two Kerberos principals, and the Like SCM, OM also requires two Kerberos principals, and the
corresponding key tab files for both of these principals. corresponding key tab files for both of these principals.
<br> <br>
<table class="table table-dark"> <table class="table table-dark">
<thead> <thead>
<tr> <tr>
<th scope="col">Property</th> <th scope="col">Property</th>
<th scope="col">Description</th> <th scope="col">Description</th>
</tr> </tr>
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<th scope="row">ozone.om.kerberos.principal</th> <td>ozone.om.kerberos.principal</th>
<td>The OzoneManager service principal. e.g. om/_HOST@REALM <td>The OzoneManager service principal. <br/> e.g. om/_HOST@REALM.COM</td>
.COM</td> </tr>
</tr> <tr>
<tr> <td>ozone.om.kerberos.keytab.file</th>
<th scope="row">ozone.om.kerberos.keytab.file</th> <td>TThe keytab file used by SCM daemon to login as its service principal.</td>
<td>TThe keytab file used by SCM daemon to login as its service principal.</td> </tr>
</tr> <tr>
<tr> <td>ozone.om.http.kerberos.principal</th>
<th scope="row">ozone.om.http.kerberos.principal</th> <td>Ozone Manager http server service principal.</td>
<td>Ozone Manager http server service principal.</td> </tr>
</tr> <tr>
<tr> <td>ozone.om.http.kerberos.keytab</th>
<th scope="row">ozone.om.http.kerberos.keytab</th> <td>The keytab file used by OM http server to login as its service principal.</td>
<td>The keytab file used by OM http server to login as its service principal.</td> </tr>
</tr> </tbody>
</tbody> </table>
</table> </div>
</div>
</div> </div>
<div class="card"> <div class="card">
<div class="card-body"> <div class="card-body">
<h3 class="card-title">S3 Gateway</h3> <h3 class="card-title">S3 Gateway</h3>
<p class="card-text"> <p class="card-text">
<br> <br>
S3 gateway requires one service principal and here the configuration values S3 gateway requires one service principal and here the configuration values
needed in the ozone-site.xml. needed in the ozone-site.xml.
<br> <br>
<table class="table table-dark"> <table class="table table-dark">
<thead> <thead>
<tr> <tr>
<th scope="col">Property</th> <th scope="col">Property</th>
<th scope="col">Description</th> <th scope="col">Description</th>
</tr> </tr>
</thead> </thead>
<tr> <tbody>
<th scope="row">ozone.s3g.keytab.file</th> <tr>
<td>The keytab file used by S3 gateway</td> <td>ozone.s3g.authentication.kerberos.principal</th>
</tr> <td>S3 Gateway principal. <br/> e.g. HTTP/_HOST@EXAMPLE.COM</td>
<tr> </tr>
<th scope="row">ozone.s3g.authentication.kerberos <tr>
.principal</th> <td>ozone.s3g.keytab.file</th>
<td>S3 Gateway principal. e.g. HTTP/_HOST@EXAMPLE.COM</td> <td>The keytab file used by S3 gateway</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
</div> </div>
</div> </div>
</div> </div>

13
hadoop-hdds/docs/content/security/SecuringDatanodes.md
View File

@ -32,10 +32,13 @@ However, we support the legacy Kerberos based Authentication to make it easy
for the current set of users.The HDFS configuration keys are the following for the current set of users.The HDFS configuration keys are the following
that is setup in hdfs-site.xml. that is setup in hdfs-site.xml.
Property|Example Value|Comment Property|Description
--------|--------------|-------------- --------|--------------
dfs.datanode.keytab.file| /keytab/dn.service.keytab| Keytab file. dfs.datanode.kerberos.principal|The datanode service principal. <br/> e.g. dn/_HOST@REALM.COM
dfs.datanode.kerberos.principal| dn/_HOST@REALM.TLD| principal name. dfs.datanode.keytab.file| The keytab file used by datanode daemon to login as its service principal.
hdds.datanode.http.kerberos.principal| Datanode http server service principal.
hdds.datanode.http.kerberos.keytab| The keytab file used by datanode http server to login as its service principal.
## How a data node becomes secure. ## How a data node becomes secure.
@ -63,7 +66,7 @@ boot time to prove the identity of the data node container (This is also work
in progress.) in progress.)
Once a certificate is issued, a Data node is secure and Ozone manager can Once a certificate is issued, a data node is secure and Ozone manager can
issue block tokens. If there is no data node certificates or the SCM's root issue block tokens. If there is no data node certificates or the SCM's root
certificate is not present in the data node, then data node will register certificate is not present in the data node, then data node will register
itself and down load the SCM's root certificate as well get the certificates itself and down load the SCM's root certificate as well get the certificates

6
hadoop-hdds/docs/content/security/SecuringS3.md
View File

@ -35,12 +35,12 @@ The user needs to `kinit` first and once they have authenticated via kerberos
* S3 clients can get the secret access id and user secret from OzoneManager. * S3 clients can get the secret access id and user secret from OzoneManager.
``` ```bash
ozone s3 getsecret ozone s3 getsecret
``` ```
This command will talk to ozone, validate the user via kerberos and generate This command will talk to ozone, validate the user via kerberos and generate
the AWS credentials. The values will be printed out on the screen. You can the AWS credentials. The values will be printed out on the screen. You can
set these values up in your .aws file for automatic access while working set these values up in your _.aws_ file for automatic access while working
against Ozone S3 buckets. against Ozone S3 buckets.
<div class="alert alert-danger" role="alert"> <div class="alert alert-danger" role="alert">
@ -51,7 +51,7 @@ against Ozone S3 buckets.
* Now you can proceed to setup these secrets in aws configs: * Now you can proceed to setup these secrets in aws configs:
``` ```bash
aws configure set default.s3.signature_version s3v4 aws configure set default.s3.signature_version s3v4
aws configure set aws_access_key_id ${accessId} aws configure set aws_access_key_id ${accessId}
aws configure set aws_secret_access_key ${secret} aws configure set aws_secret_access_key ${secret}

5
hadoop-hdds/docs/content/security/SecuringTDE.md
View File

@ -22,20 +22,19 @@ icon: lock
limitations under the License. limitations under the License.
--> -->
## Transparent Data Encryption
Ozone TDE setup process and usage are very similar to HDFS TDE. Ozone TDE setup process and usage are very similar to HDFS TDE.
The major difference is that Ozone TDE is enabled at Ozone bucket level The major difference is that Ozone TDE is enabled at Ozone bucket level
when a bucket is created. when a bucket is created.
### Setting up the Key Management Server ### Setting up the Key Management Server
To use TDE, clients must setup a Key Management server and provide that URI to To use TDE, clients must setup a Key Management Server and provide that URI to
Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this
configuration can be provided via *hdfs-site.xml*. configuration can be provided via *hdfs-site.xml*.
Property| Value Property| Value
-----------------------------------|----------------------------------------- -----------------------------------|-----------------------------------------
hadoop.security.key.provider.path | KMS uri. e.g. kms://http@kms-host:9600/kms hadoop.security.key.provider.path | KMS uri. <br> e.g. kms://http@kms-host:9600/kms
### Using Transparent Data Encryption ### Using Transparent Data Encryption
If this is already configured for your cluster, then you can simply proceed If this is already configured for your cluster, then you can simply proceed

35
hadoop-hdds/docs/content/security/SecurityAcls.md
View File

@ -21,9 +21,9 @@ summary: Native ACL support provides ACL functionality without Ranger integratio
limitations under the License. limitations under the License.
--> -->
Ozone supports a set of native ACLs. These ACLs cane be used independently or Ozone supports a set of native ACLs. These ACLs can be used independently or
along with Ranger. If Apache Ranger is enabled, then ACL will be checked along with Ranger. If Apache Ranger is enabled, then ACL will be checked
first with Ranger and then Ozone's internal ACLs will be evaluated. first with Ranger and then Ozone's internal ACLs will be evaluated.
Ozone ACLs are a super set of Posix and S3 ACLs. Ozone ACLs are a super set of Posix and S3 ACLs.
@ -31,10 +31,10 @@ The general format of an ACL is _object_:_who_:_rights_.
Where an _object_ can be: Where an _object_ can be:
1. **Volume** - An Ozone volume. e.g. /volume 1. **Volume** - An Ozone volume. e.g. _/volume_
2. **Bucket** - An Ozone bucket. e.g. /volume/bucket 2. **Bucket** - An Ozone bucket. e.g. _/volume/bucket_
3. **Key** - An object key or an object. e.g. /volume/bucket/key 3. **Key** - An object key or an object. e.g. _/volume/bucket/key_
4. **Prefix** - A path prefix for a specific key. e.g. /volume/bucket/prefix1/prefix2 4. **Prefix** - A path prefix for a specific key. e.g. _/volume/bucket/prefix1/prefix2_
Where a _who_ can be: Where a _who_ can be:
@ -63,23 +63,20 @@ volume and keys in a bucket. Please note: Under Ozone, Only admins can create vo
to the volume and buckets which allow listing of the child objects. Please note: The user and admins can list the volumes owned by the user. to the volume and buckets which allow listing of the child objects. Please note: The user and admins can list the volumes owned by the user.
3. **Delete** – Allows the user to delete a volume, bucket or key. 3. **Delete** – Allows the user to delete a volume, bucket or key.
4. **Read** – Allows the user to read the metadata of a Volume and Bucket and 4. **Read** – Allows the user to read the metadata of a Volume and Bucket and
data stream and metadata of a key(object). data stream and metadata of a key.
5. **Write** - Allows the user to write the metadata of a Volume and Bucket and 5. **Write** - Allows the user to write the metadata of a Volume and Bucket and
allows the user to overwrite an existing ozone key(object). allows the user to overwrite an existing ozone key.
6. **Read_ACL** – Allows a user to read the ACL on a specific object. 6. **Read_ACL** – Allows a user to read the ACL on a specific object.
7. **Write_ACL** – Allows a user to write the ACL on a specific object. 7. **Write_ACL** – Allows a user to write the ACL on a specific object.
<h3>Ozone Native ACL APIs <span class="badge badge-secondary">Work in <h3>Ozone Native ACL APIs</h3>
progress</span></h3>
The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs
supported are: supported are:
1. **SetAcl** – This API will take user principal, the name of the object, type 1. **SetAcl** – This API will take user principal, the name, type
of the object and a list of ACLs. of the ozone object and a list of ACLs.
2. **GetAcl** – This API will take the name and type of the ozone object
2. **GetAcl** – This API will take the name of an ozone object and type of the and will return a list of ACLs.
object and will return a list of ACLs. 3. **RemoveAcl** - This API will take the name, type of the
3. **RemoveAcl** - It is possible that we might support an API called RemoveACL ozone object and the ACL that has to be removed.
as a convenience API, but in reality it is just a GetACL followed by SetACL
with an etag to avoid conflicts.

30
hadoop-hdds/docs/content/shell/BucketCommands.md
View File

@ -29,7 +29,7 @@ Ozone shell supports the following bucket commands.
### Create ### Create
The bucket create command allows users to create a bucket. The `bucket create` command allows users to create a bucket.
***Params:*** ***Params:***
@ -46,7 +46,7 @@ Since no scheme was specified this command defaults to O3 (RPC) protocol.
### Delete ### Delete
The bucket delete command allows users to delete a bucket. If the The `bucket delete` command allows users to delete a bucket. If the
bucket is not empty then this command will fail. bucket is not empty then this command will fail.
***Params:*** ***Params:***
@ -63,7 +63,8 @@ The above command will delete _jan_ bucket if it is empty.
### Info ### Info
The bucket info commands returns the information about the bucket. The `bucket info` commands returns the information about the bucket.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
@ -78,15 +79,15 @@ The above command will print out the information about _jan_ bucket.
### List ### List
The bucket list command allows users to list the buckets in a volume. The `bucket list` command allows users to list the buckets in a volume.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
|--------------------------------|-----------------------------------------| |--------------------------------|-----------------------------------------|
| -l, --length | Maximum number of results to return. Default: 100 | -l, \-\-length | Maximum number of results to return. Default: 100
| -p, --prefix | Optional, Only buckets that match this prefix will be returned. | -p, \-\-prefix | Optional, Only buckets that match this prefix will be returned.
| -s, --start | The listing will start from key after the start key. | -s, \-\-start | The listing will start from key after the start key.
| Uri | The name of the _volume_. | Uri | The name of the _volume_.
{{< highlight bash >}} {{< highlight bash >}}
@ -94,18 +95,3 @@ ozone sh bucket list /hive
{{< /highlight >}} {{< /highlight >}}
This command will list all buckets on the volume _hive_. This command will list all buckets on the volume _hive_.
### path
The bucket command to provide ozone mapping for s3 bucket (Created via aws cli)
{{< highlight bash >}}
ozone s3 path <<s3Bucket>>
{{< /highlight >}}
The above command will print VolumeName and the mapping created for s3Bucket.
You can try out these commands from the docker instance of the [Alpha
Cluster](runningviadocker.html).

24
hadoop-hdds/docs/content/shell/KeyCommands.md
View File

@ -34,7 +34,7 @@ Ozone shell supports the following key commands.
### Get ### Get
The key get command downloads a key from Ozone cluster to local file system. The `key get` command downloads a key from Ozone cluster to local file system.
***Params:*** ***Params:***
@ -52,7 +52,7 @@ local file sales.orc.
### Put ### Put
Uploads a file from the local file system to the specified bucket. The `key put` command uploads a file from the local file system to the specified bucket.
***Params:*** ***Params:***
@ -61,7 +61,7 @@ Uploads a file from the local file system to the specified bucket.
|--------------------------------|-----------------------------------------| |--------------------------------|-----------------------------------------|
| Uri | The name of the key in **/volume/bucket/key** format. | Uri | The name of the key in **/volume/bucket/key** format.
| FileName | Local file to upload. | FileName | Local file to upload.
| -r, --replication | Optional, Number of copies, ONE or THREE are the options. Picks up the default from cluster configuration. | -r, \-\-replication | Optional, Number of copies, ONE or THREE are the options. Picks up the default from cluster configuration.
{{< highlight bash >}} {{< highlight bash >}}
ozone sh key put /hive/jan/corrected-sales.orc sales.orc ozone sh key put /hive/jan/corrected-sales.orc sales.orc
@ -70,7 +70,7 @@ The above command will put the sales.orc as a new key into _/hive/jan/corrected-
### Delete ### Delete
The key delete command removes the key from the bucket. The `key delete` command removes the key from the bucket.
***Params:*** ***Params:***
@ -87,7 +87,8 @@ The above command deletes the key _/hive/jan/corrected-sales.orc_.
### Info ### Info
The key info commands returns the information about the key. The `key info` commands returns the information about the key.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
@ -103,15 +104,15 @@ key.
### List ### List
The key list command allows user to list all keys in a bucket. The `key list` command allows user to list all keys in a bucket.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
|--------------------------------|-----------------------------------------| |--------------------------------|-----------------------------------------|
| -l, --length | Maximum number of results to return. Default: 1000 | -l, \-\-length | Maximum number of results to return. Default: 1000
| -p, --prefix | Optional, Only buckets that match this prefix will be returned. | -p, \-\-prefix | Optional, Only buckets that match this prefix will be returned.
| -s, --start | The listing will start from key after the start key. | -s, \-\-start | The listing will start from key after the start key.
| Uri | The name of the _volume_. | Uri | The name of the _volume_.
{{< highlight bash >}} {{< highlight bash >}}
@ -135,7 +136,4 @@ The `key rename` command changes the name of an existing key in the specified bu
{{< highlight bash >}} {{< highlight bash >}}
ozone sh key rename /hive/jan sales.orc new_name.orc ozone sh key rename /hive/jan sales.orc new_name.orc
{{< /highlight >}} {{< /highlight >}}
The above command will rename `sales.orc` to `new_name.orc` in the bucket `/hive/jan`. The above command will rename _sales.orc_ to _new\_name.orc_ in the bucket _/hive/jan_.
You can try out these commands from the docker instance of the [Alpha
Cluster](runningviadocker.html).

20
hadoop-hdds/docs/content/shell/VolumeCommands.md
View File

@ -30,15 +30,15 @@ Volume commands generally need administrator privileges. The ozone shell support
### Create ### Create
The volume create command allows an administrator to create a volume and The `volume create` command allows an administrator to create a volume and
assign it to a user. assign it to a user.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
|--------------------------------|-----------------------------------------| |--------------------------------|-----------------------------------------|
| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | | -q, \-\-quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. |
| -u, --user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | | -u, \-\-user | Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume. |
| Uri | The name of the volume. | | Uri | The name of the volume. |
{{< highlight bash >}} {{< highlight bash >}}
@ -50,7 +50,7 @@ volume has a quota of 1TB, and the owner is _bilbo_.
### Delete ### Delete
The volume delete command allows an administrator to delete a volume. If the The `volume delete` command allows an administrator to delete a volume. If the
volume is not empty then this command will fail. volume is not empty then this command will fail.
***Params:*** ***Params:***
@ -68,8 +68,9 @@ inside it.
### Info ### Info
The volume info commands returns the information about the volume including The `volume info` commands returns the information about the volume including
quota and owner information. quota and owner information.
***Params:*** ***Params:***
| Arguments | Comment | | Arguments | Comment |
@ -84,7 +85,7 @@ The above command will print out the information about hive volume.
### List ### List
The volume list command will list the volumes owned by a user. The `volume list` command will list the volumes owned by a user.
{{< highlight bash >}} {{< highlight bash >}}
ozone sh volume list --user hadoop ozone sh volume list --user hadoop
@ -100,8 +101,8 @@ The volume update command allows changing of owner and quota on a given volume.
| Arguments | Comment | | Arguments | Comment |
|--------------------------------|-----------------------------------------| |--------------------------------|-----------------------------------------|
| -q, --quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. | | -q, \-\-quota | Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster. |
| -u, --user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. | | -u, \-\-user | Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume. |
| Uri | The name of the volume. | | Uri | The name of the volume. |
{{< highlight bash >}} {{< highlight bash >}}
@ -109,6 +110,3 @@ ozone sh volume update --quota=10TB /hive
{{< /highlight >}} {{< /highlight >}}
The above command updates the volume quota to 10TB. The above command updates the volume quota to 10TB.
You can try out these commands from the docker instance of the [Alpha
Cluster](runningviadocker.html).

2
hadoop-hdds/docs/content/start/Kubernetes.md
View File

@ -25,7 +25,7 @@ title: Ozone on Kubernetes
{{< /requirements >}} {{< /requirements >}}
As the _apache/ozone_ docker images are available from the dockerhub the deployment process is very similar Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) As the _apache/ozone_ docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host)
Deploy to kubernetes Deploy to kubernetes
`kubernetes/examples` folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. `kubernetes/examples` folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases.

7
hadoop-hdds/docs/content/start/OnPrem.md
View File

@ -33,7 +33,7 @@ requests blocks from SCM, to which clients can write data.
## Setting up an Ozone only cluster ## Setting up an Ozone only cluster
* Please untar the ozone-<version> to the directory where you are going * Please untar the ozone-\<version\> to the directory where you are going
to run Ozone from. We need Ozone jars on all machines in the cluster. So you to run Ozone from. We need Ozone jars on all machines in the cluster. So you
need to do this on all machines in the cluster. need to do this on all machines in the cluster.
@ -152,14 +152,13 @@ ozone om --init
{{< /highlight >}} {{< /highlight >}}
Once Ozone manager has created the Object Store, we are ready to run the name Once Ozone manager is initialized, we are ready to run the name service.
services.
{{< highlight bash >}} {{< highlight bash >}}
ozone --daemon start om ozone --daemon start om
{{< /highlight >}} {{< /highlight >}}
At this point Ozone's name services, the Ozone manager, and the block service SCM is both running. At this point Ozone's name services, the Ozone manager, and the block service SCM is both running.\
**Please note**: If SCM is not running **Please note**: If SCM is not running
```om --init``` command will fail. SCM start will fail if on-disk data structures are missing. So please make sure you have done both ```scm --init``` and ```om --init``` commands. ```om --init``` command will fail. SCM start will fail if on-disk data structures are missing. So please make sure you have done both ```scm --init``` and ```om --init``` commands.

7
hadoop-hdds/docs/content/start/StartFromDockerHub.md
View File

@ -30,7 +30,7 @@ The easiest way to start up an all-in-one ozone container is to use the latest
docker image from docker hub: docker image from docker hub:
```bash ```bash
docker run -P 9878:9878 -P 9876:9876 apache/ozone docker run -p 9878:9878 -p 9876:9876 apache/ozone
``` ```
This command will pull down the ozone image from docker hub and start all This command will pull down the ozone image from docker hub and start all
ozone services in a single container. <br> ozone services in a single container. <br>
@ -40,7 +40,7 @@ Container Manager) one data node and the S3 compatible REST server
# Local multi-container cluster # Local multi-container cluster
If you would like to use a more realistic pseud-cluster where each components If you would like to use a more realistic pseudo-cluster where each components
run in own containers, you can start it with a docker-compose file. run in own containers, you can start it with a docker-compose file.
We have shipped a docker-compose and an enviorment file as part of the We have shipped a docker-compose and an enviorment file as part of the
@ -65,7 +65,7 @@ If you need multiple datanodes, we can just scale it up:
``` ```
# Running S3 Clients # Running S3 Clients
Once the cluster is booted up and ready, you can verify it is running by Once the cluster is booted up and ready, you can verify its status by
connecting to the SCM's UI at [http://localhost:9876](http://localhost:9876). connecting to the SCM's UI at [http://localhost:9876](http://localhost:9876).
The S3 gateway endpoint will be exposed at port 9878. You can use Ozone's S3 The S3 gateway endpoint will be exposed at port 9878. You can use Ozone's S3
@ -103,7 +103,6 @@ our bucket.
aws s3 --endpoint http://localhost:9878 ls s3://bucket1/testfile aws s3 --endpoint http://localhost:9878 ls s3://bucket1/testfile
``` ```
.
<div class="alert alert-info" role="alert"> You can also check the internal <div class="alert alert-info" role="alert"> You can also check the internal
bucket browser supported by Ozone S3 interface by clicking on the below link. bucket browser supported by Ozone S3 interface by clicking on the below link.
<br> <br>