131 lines
4.1 KiB
Markdown
131 lines
4.1 KiB
Markdown
|
---
|
||
|
|
title: S3
|
||
|
|
menu:
|
||
|
|
main:
|
||
|
|
parent: Client
|
||
|
|
weight: 1
|
||
|
|
---
|
||
|
|
|
||
|
|
<!---
|
||
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
||
|
|
contributor license agreements. See the NOTICE file distributed with
|
||
|
|
this work for additional information regarding copyright ownership.
|
||
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
||
|
|
(the "License"); you may not use this file except in compliance with
|
||
|
|
the License. You may obtain a copy of the License at
|
||
|
|
|
||
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
||
|
|
|
||
|
|
Unless required by applicable law or agreed to in writing, software
|
||
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||
|
|
See the License for the specific language governing permissions and
|
||
|
|
limitations under the License.
|
||
|
|
-->
|
||
|
|
|
||
|
|
Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.
|
||
|
|
|
||
|
|
## Getting started
|
||
|
|
|
||
|
|
S3 Gateway is a separated component which provides the S3 compatible. It should be started additional to the regular Ozone components.
|
||
|
|
|
||
|
|
You can start a docker based cluster, including the S3 gateway from the release package.
|
||
|
|
|
||
|
|
Go to the `compose/ozones3` directory, and start the server:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
docker-compose up -d
|
||
|
|
```
|
||
|
|
|
||
|
|
You can access the S3 gateway at `http://localhost:9878`
|
||
|
|
|
||
|
|
## URL Schema
|
||
|
|
|
||
|
|
Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname)
|
||
|
|
|
||
|
|
By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your `ozone-site.xml`:
|
||
|
|
|
||
|
|
```xml
|
||
|
|
<property>
|
||
|
|
<name>ozone.s3g.domain.name</name>
|
||
|
|
<value>s3g.internal</value>
|
||
|
|
</property>
|
||
|
|
```
|
||
|
|
|
||
|
|
## Bucket browser
|
||
|
|
|
||
|
|
Bucket's could be browsed from the browser with adding `?browser=true` to the bucket URL.
|
||
|
|
|
||
|
|
For example the content of the 'testbucket' could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true
|
||
|
|
|
||
|
|
|
||
|
|
## Implemented REST endpoints
|
||
|
|
|
||
|
|
Operations on S3Gateway service:
|
||
|
|
|
||
|
|
Endpoint | Status |
|
||
|
|
------------|-------------|
|
||
|
|
GET service | implemented |
|
||
|
|
|
||
|
|
Operations on Bucket:
|
||
|
|
|
||
|
|
Endpoint | Status | Notes
|
||
|
|
------------------------------------|-------------|---------------
|
||
|
|
GET Bucket (List Objects) Version 2 | implemented |
|
||
|
|
HEAD Bucket | implemented |
|
||
|
|
DELETE Bucket | implemented |
|
||
|
|
PUT Bucket (Create bucket) | implemented |
|
||
|
|
Delete Multiple Objects (POST) | implemented |
|
||
|
|
|
||
|
|
Operation on Objects:
|
||
|
|
|
||
|
|
Endpoint | Status | Notes
|
||
|
|
------------------------------------|-----------------|---------------
|
||
|
|
PUT Object | implemented |
|
||
|
|
GET Object | implemented | Range headers are not supported
|
||
|
|
Multipart Uplad | not implemented |
|
||
|
|
DELETE Object | implemented |
|
||
|
|
HEAD Object | implemented |
|
||
|
|
|
||
|
|
|
||
|
|
## Security
|
||
|
|
|
||
|
|
Security is not yet implemented, you can *use* any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
|
||
|
|
|
||
|
|
Note: Ozone has a notion for 'volumes' which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are maped to Ozone 'volume/bucket' locations (depending from the given authentication information).
|
||
|
|
|
||
|
|
To show the storage location of a S3 bucket, use the `ozone sh bucket path <bucketname>` command.
|
||
|
|
|
||
|
|
```
|
||
|
|
aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket=bucket1
|
||
|
|
|
||
|
|
ozone sh bucket path bucket1
|
||
|
|
Volume name for S3Bucket is : s3thisisakey
|
||
|
|
Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
|
||
|
|
```
|
||
|
|
|
||
|
|
## Clients
|
||
|
|
|
||
|
|
### AWS Cli
|
||
|
|
|
||
|
|
`aws` CLI could be used with specifying the custom REST endpoint.
|
||
|
|
|
||
|
|
```
|
||
|
|
aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
|
||
|
|
```
|
||
|
|
|
||
|
|
Or
|
||
|
|
|
||
|
|
```
|
||
|
|
aws s3 ls --endpoint http://localhost:9878 s3://buckettest
|
||
|
|
```
|
||
|
|
|
||
|
|
### S3 Fuse driver (goofys)
|
||
|
|
|
||
|
|
Goofys is a S3 FUSE driver. It could be used to any mount any Ozone bucket as posix file system:
|
||
|
|
|
||
|
|
|
||
|
|
```
|
||
|
|
goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
|
||
|
|
```
|