diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md index 3b9b5c4912..19d322db87 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md @@ -76,13 +76,10 @@ over long-haul connections. Please help us identify and fix these problems ## Setting up the tests -To integration test the S3* filesystem clients, you need to provide two files -which pass in authentication details to the test runner. +To integration test the S3* filesystem clients, you need to provide +`auth-keys.xml` which passes in authentication details to the test runner. -1. `auth-keys.xml` -1. `contract-test-options.xml` - -These are both Hadoop XML configuration files, which must be placed into +It is a Hadoop XML configuration file, which must be placed into `hadoop-tools/hadoop-aws/src/test/resources`. ### File `core-site.xml` @@ -106,6 +103,8 @@ each filesystem for its testing. 1. `test.fs.s3n.name` : the URL of the bucket for S3n tests 1. `test.fs.s3a.name` : the URL of the bucket for S3a tests +1. `fs.contract.test.fs.s3n` : the URL of the bucket for S3n filesystem contract tests +1. `fs.contract.test.fs.s3a` : the URL of the bucket for S3a filesystem contract tests *Note* that running s3a and s3n tests in parallel mode, against the same bucket is unreliable. We recommend using separate buckets or testing one connector @@ -128,8 +127,8 @@ Example: - test.fs.s3a.name - s3a://test-aws-s3a/ + fs.contract.test.fs.s3n + ${test.fs.s3n.name} @@ -142,6 +141,16 @@ Example: DONOTEVERSHARETHISSECRETKEY! + + test.fs.s3a.name + s3a://test-aws-s3a/ + + + + fs.contract.test.fs.s3a + ${test.fs.s3a.name} + + fs.s3a.access.key AWS access key ID. Omit for IAM role-based authentication. @@ -163,107 +172,6 @@ Example: ``` -### File `contract-test-options.xml` - -The file `hadoop-tools/hadoop-aws/src/test/resources/contract-test-options.xml` -must be created and configured for the test filesystems. - -If a specific file `fs.contract.test.fs.*` test path is not defined for -any of the filesystems, those tests will be skipped. - -The standard S3 authentication details must also be provided. This can be -through copy-and-paste of the `auth-keys.xml` credentials, or it can be -through direct XInclude inclusion. - -Here is an an example `contract-test-options.xml` which places all test options -into the `auth-keys.xml` file, so offering a single place to keep credentials -and define test endpoint bindings. - -```xml - - - -``` - -### s3n:// - - -In the file `src/test/resources/contract-test-options.xml`, the filesystem -name must be defined in the property `fs.contract.test.fs.s3n`. -The standard configuration options to define the S3N authentication details -must also be provided. - -Example: - -```xml - - fs.contract.test.fs.s3n - s3n://test-aws-s3n/ - -``` - -### s3a:// - - -In the file `src/test/resources/contract-test-options.xml`, the filesystem -name must be defined in the property `fs.contract.test.fs.s3a`. -The standard configuration options to define the S3N authentication details -must also be provided. - -Example: - -```xml - - fs.contract.test.fs.s3a - s3a://test-aws-s3a/ - -``` - -### Complete example of `contract-test-options.xml` - - -```xml - - - - - - - - - - fs.contract.test.fs.s3a - s3a://test-aws-s3a/ - - - - fs.contract.test.fs.s3n - s3n://test-aws-s3n/ - - - -``` - -This example pulls in the `auth-keys.xml` file for the credentials. -This provides one single place to keep the keys up to date —and means -that the file `contract-test-options.xml` does not contain any -secret credentials itself. As the auth keys XML file is kept out of the -source code tree, it is not going to get accidentally committed. - ### Configuring S3a Encryption For S3a encryption tests to run correctly, the @@ -346,8 +254,7 @@ like `ITestS3A*` shown above, it may cause unpredictable test failures. ### Testing against different regions S3A can connect to different regions —the tests support this. Simply -define the target region in `contract-test-options.xml` or any `auth-keys.xml` -file referenced. +define the target region in `auth-keys.xml`. ```xml