diff --git a/hadoop-ozone/docs/content/S3.md b/hadoop-ozone/docs/content/S3.md new file mode 100644 index 0000000000..cfefaf415e --- /dev/null +++ b/hadoop-ozone/docs/content/S3.md @@ -0,0 +1,130 @@ +--- +title: S3 +menu: + main: + parent: Client + weight: 1 +--- + + + +Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools. + +## Getting started + +S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components. + +You can start a docker based cluster, including the S3 gateway from the release package. + +Go to the `compose/ozones3` directory, and start the server: + +```bash +docker-compose up -d +``` + +You can access the S3 gateway at `http://localhost:9878` + +## URL Schema + +Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname) + +By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`: + +```xml + + ozone.s3g.domain.name + s3g.internal + +``` + +## Bucket browser + +Bucket's could be browsed from the browser with adding `?browser=true` to the bucket URL. + +For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true + + +## Implemented REST endpoints + +Operations on S3Gateway service: + +Endpoint | Status | +------------|-------------| +GET service | implemented | + +Operations on Bucket: + +Endpoint | Status | Notes +------------------------------------|-------------|--------------- +GET Bucket (List Objects) Version 2 | implemented | +HEAD Bucket | implemented | +DELETE Bucket | implemented | +PUT Bucket (Create bucket) | implemented | +Delete Multiple Objects (POST) | implemented | + +Operation on Objects: + +Endpoint | Status | Notes +------------------------------------|-----------------|--------------- +PUT Object | implemented | +GET Object | implemented | Range headers are not supported +Multipart Uplad | not implemented | +DELETE Object | implemented | +HEAD Object | implemented | + + +## Security + +Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY + +Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are maped to Ozone 'volume/bucket' locations (depending from the given authentication information). + +To show the storage location of a S3 bucket, use the `ozone sh bucket path ` command. + +``` +aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1 + +ozone sh bucket path bucket1 +Volume name for S3Bucket is : s3thisisakey +Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey +``` + +## Clients + +### AWS Cli + +`aws` CLI could be used with specifying the custom REST endpoint. + +``` +aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest +``` + +Or + +``` +aws s3 ls --endpoint http://localhost:9878 s3://buckettest +``` + +### S3 Fuse driver (goofys) + +Goofys is a S3 FUSE driver. It could be used to any mount any Ozone bucket as posix file system: + + +``` +goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1 +```