diff --git a/hadoop-hdds/docs/content/BucketCommands.md b/hadoop-hdds/docs/content/BucketCommands.md index 61537628fb..c726f6d82f 100644 --- a/hadoop-hdds/docs/content/BucketCommands.md +++ b/hadoop-hdds/docs/content/BucketCommands.md @@ -2,8 +2,8 @@ title: Bucket Commands menu: main: - parent: Client - weight: 3 + parent: OzoneShell + weight: 2 --- -[Prometheus](https://prometheus.io/) is an open-source monitoring server developed under under the [Cloud Native Foundation](Cloud Native Foundation). +[Prometheus](https://prometheus.io/) is an open-source monitoring server developed under under the [Cloud Native Computing Foundation](https://www.cncf.io/). Ozone supports Prometheus out of the box. The servers start a prometheus compatible metrics endpoint where all the available hadoop metrics are published in prometheus exporter format. @@ -75,14 +75,14 @@ prometheus http://localhost:9090/targets -![Prometheus target page example](../../prometheus.png) +![Prometheus target page example](prometheus.png) (6) Check any metrics on the prometheus web ui. For example: http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1 -![Prometheus target page example](../../prometheus-key-allocate.png) +![Prometheus target page example](prometheus-key-allocate.png) ## Note diff --git a/hadoop-hdds/docs/content/RunningViaDocker.md b/hadoop-hdds/docs/content/RunningViaDocker.md index 0b8fece5fb..e60886f0c8 100644 --- a/hadoop-hdds/docs/content/RunningViaDocker.md +++ b/hadoop-hdds/docs/content/RunningViaDocker.md @@ -44,16 +44,16 @@ including the data nodes and ozone services. ozone instance on your machine. {{< highlight bash >}} -cd ozone-0.2.1-SNAPSHOT/compose/ozone/ +cd compose/ozone/ docker-compose up -d {{< /highlight >}} - To verify that ozone is working as expected, let us log into a data node and run _freon_, the load generator for Ozone. The ```exec datanode bash``` command -will open a bash shell on the datanode. The ozone freon command is executed -within the datanode container. You can quit freon via CTRL-C any time. The +will open a bash shell on the datanode. + +The `ozone freon` command is executed within the datanode container. You can quit freon via CTRL-C any time. The ```rk``` profile instructs freon to generate random keys. {{< highlight bash >}} diff --git a/hadoop-hdds/docs/content/S3.md b/hadoop-hdds/docs/content/S3.md index bca2d3001c..a9281649c9 100644 --- a/hadoop-hdds/docs/content/S3.md +++ b/hadoop-hdds/docs/content/S3.md @@ -83,16 +83,37 @@ Endpoint | Status | Notes ------------------------------------|-----------------|--------------- PUT Object | implemented | GET Object | implemented | Range headers are not supported -Multipart Uplad | not implemented | +Multipart Uplad | implemented |Except the listing of the current MultiPartUploads. DELETE Object | implemented | HEAD Object | implemented | ## Security -Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY +If security is not enabled, you can *use* **any** AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY -Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone 'volume/bucket' locations (depending on the given authentication information). +If security is enabled, you can get the key and the secret with the `ozone s3 getsecret` command (*kerberos based authentication is required). + +``` +/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM +ozone s3 getsecret +awsAccessKey=testuser/scm@EXAMPLE.COM +awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999 + +``` + +Now, you can use the key and the secret to access the S3 endpoint: + +``` +export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM +export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999 +aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1 +``` + + +## S3 bucket name mapping to Ozone buckets + +**Note**: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone 'volume/bucket' locations (depending on the given authentication information). To show the storage location of a S3 bucket, use the `ozone s3 path ` command. diff --git a/hadoop-hdds/docs/content/S3Commands.md b/hadoop-hdds/docs/content/S3Commands.md deleted file mode 100644 index 23936e6d04..0000000000 --- a/hadoop-hdds/docs/content/S3Commands.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: S3 Commands -menu: - main: - parent: Client - weight: 3 ---- - - -Ozone shell supports the following S3 commands. - - * [getsecret](#get secret) - -### Get Secret - -User should get the kerberos ticket before using this option. - - -{{< highlight bash >}} -ozone s3 getsecret -{{< /highlight >}} -Prints the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID for the current user. - - -You can try out these commands from the docker instance of the [Alpha -Cluster](runningviadocker.html). diff --git a/hadoop-hdds/docs/content/Settings.md b/hadoop-hdds/docs/content/Settings.md index b02816b839..c5d01df11c 100644 --- a/hadoop-hdds/docs/content/Settings.md +++ b/hadoop-hdds/docs/content/Settings.md @@ -71,7 +71,7 @@ Here is an example, {{< highlight xml >}} ozone.enabled - True + true {{< /highlight >}} @@ -107,7 +107,7 @@ Here is an example, * **ozone.scm.datanode.id** Data nodes generate a Unique ID called Datanode ID. This identity is written to the file specified by this path. *Data nodes - will create this path if it doesn't exist already.* + will create this path if it doesn't exist already.* Here is an example, {{< highlight xml >}} diff --git a/hadoop-hdds/docs/content/SetupSecureOzone.md b/hadoop-hdds/docs/content/SetupSecureOzone.md index f003becf45..e67c1e76c0 100644 --- a/hadoop-hdds/docs/content/SetupSecureOzone.md +++ b/hadoop-hdds/docs/content/SetupSecureOzone.md @@ -25,8 +25,10 @@ weight: 11 # Setup secure ozone cluster # To enable security in ozone cluster **ozone.security.enabled** should be set to true. -ozone.security.enabled| true +Property|Value ----------------------|------ +ozone.security.enabled| true + ## Kerberos ## Configuration for service daemons: @@ -45,20 +47,28 @@ ozone.s3g.authentication.kerberos.principal|S3 Gateway principal. Ex HTTP/_HOST@ ## Tokens ## ## Delegation token ## + Delegation tokens are enabled by default when security is enabled. ## Block Tokens ## -hdds.block.token.enabled | true + +Property|Value -----------------------------|------ +hdds.block.token.enabled | true ## S3Token ## + S3 token are enabled by default when security is enabled. To use S3 tokens users need to perform following steps: + * S3 clients should get the secret access id and user secret from OzoneManager. + ``` ozone s3 getsecret ``` + * Setup secret in aws configs: + ``` aws configure set default.s3.signature_version s3v4 aws configure set aws_access_key_id ${accessId} @@ -67,18 +77,22 @@ aws configure set region us-west-1 ``` ## Certificates ## + Certificates are used internally inside Ozone. Its enabled be default when security is enabled. ## Authorization ## + Default access authorizer for Ozone approves every request. It is not suitable for production environments. It is recommended that clients use ranger plugin for Ozone to manage authorizations. -Property|Description +Property|Value --------|------------------------------------------------------------ ozone.acl.enabled | true ozone.acl.authorizer.class| org.apache.ranger.authorization.ozone.authorizer.RangerOzoneAuthorizer ## TDE ## + To use TDE clients must set KMS URI. +Property|Value +-----------------------------------|----------------------------------------- hadoop.security.key.provider.path | KMS uri. Ex kms://http@kms-host:9600/kms ------------------------------------|----------------------------------------- \ No newline at end of file diff --git a/hadoop-hdds/docs/content/VolumeCommands.md b/hadoop-hdds/docs/content/VolumeCommands.md index fecda40166..4cdb212e7d 100644 --- a/hadoop-hdds/docs/content/VolumeCommands.md +++ b/hadoop-hdds/docs/content/VolumeCommands.md @@ -2,8 +2,8 @@ title: Volume Commands menu: main: - parent: Client - weight: 2 + parent: OzoneShell + weight: 1 ---