hadoop/hadoop-hdds/docs/content/S3.md

---
title: S3
menu:
   main:
      parent: Client
      weight: 1
---

<!---
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.

## Getting started

S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.

You can start a docker based cluster, including the S3 gateway from the release package.

Go to the `compose/ozones3` directory, and start the server:

```bash
docker-compose up -d
```

You can access the S3 gateway at `http://localhost:9878`

## URL Schema

Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)

By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`:

```xml
<property>
   <name>ozone.s3g.domain.name</name>
   <value>s3g.internal</value>
</property>
```

## Bucket browser

Buckets could be browsed from the browser by adding `?browser=true` to the bucket URL.

For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true


## Implemented REST endpoints

Operations on S3Gateway service:

Endpoint    | Status      |
------------|-------------|
GET service | implemented |

Operations on Bucket:

Endpoint                            | Status      | Notes
------------------------------------|-------------|---------------
GET Bucket (List Objects) Version 2 | implemented |
HEAD Bucket                         | implemented |
DELETE Bucket                       | implemented |
PUT Bucket (Create bucket)          | implemented |
Delete Multiple Objects (POST)      | implemented |

Operation on Objects:

Endpoint                            | Status          | Notes
------------------------------------|-----------------|---------------
PUT Object                          | implemented     |
GET Object                          | implemented     | Range headers are not supported
Multipart Uplad                     | not implemented |
DELETE Object                       | implemented     |
HEAD Object                         | implemented     |


## Security

Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone 'volume/bucket' locations (depending on the given authentication information).

To show the storage location of a S3 bucket, use the `ozone s3 path <bucketname>` command.

```
aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1

ozone sh bucket path bucket1
Volume name for S3Bucket is : s3thisisakey
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
```

## Clients

### AWS Cli

`aws` CLI could be used with specifying the custom REST endpoint.

```
aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
```

Or

```
aws s3 ls --endpoint http://localhost:9878 s3://buckettest
```

### S3 Fuse driver (goofys)

Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system:


```
goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
```