big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.

Browse Source
This commit is contained in:
Steve Loughran 2016-06-06 23:40:49 +02:00
parent 4a1cedc010
commit a3f78d8fa8
2 changed files with 21 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View File

@ -31,6 +31,7 @@
import java.util.concurrent.ExecutorService; import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
import com.amazonaws.AmazonClientException; import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException; import com.amazonaws.AmazonServiceException;
import com.amazonaws.ClientConfiguration; import com.amazonaws.ClientConfiguration;
@ -464,6 +465,7 @@ private AWSCredentialsProvider getAWSCredentialsProvider(URI binding,
new BasicAWSCredentialsProvider( new BasicAWSCredentialsProvider(
creds.getAccessKey(), creds.getAccessSecret()), creds.getAccessKey(), creds.getAccessSecret()),
new InstanceProfileCredentialsProvider(), new InstanceProfileCredentialsProvider(),
new EnvironmentVariableCredentialsProvider(),
new AnonymousAWSCredentialsProvider() new AnonymousAWSCredentialsProvider()
); );

19
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
View File

@ -202,6 +202,25 @@ credentials in S3AFileSystem.
For additional reading on the credential provider API see: For additional reading on the credential provider API see:
[Credential Provider API](../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html). [Credential Provider API](../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html).
#### Authenticating via environment variables
S3A supports configuration via [the standard AWS environment variables](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment).
The core environment variables are for the access key and associated secret:
```
export AWS_ACCESS_KEY_ID=my.aws.key
export AWS_SECRET_ACCESS_KEY=my.secret.key
```
These environment variables can be used to set the authentication credentials
instead of properties in the Hadoop configuration. *Important:* these
environment variables are not propagated from client to server when
YARN applications are launched. That is: having the AWS environment variables
set when an application is launched will not permit the launched application
to access S3 resources. The environment variables must (somehow) be set
on the hosts/processes where the work is executed.
##### End to End Steps for Distcp and S3 with Credential Providers ##### End to End Steps for Distcp and S3 with Credential Providers
###### provision ###### provision