big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)

Browse Source 
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
This commit is contained in:
Gabor Bota 2021-01-05 15:43:01 +01:00 committed by GitHub
parent 2b4febcf57
commit 42eb9ff68e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
View File

@ -499,7 +499,7 @@ private Constants() {
* will be validated using {@code S3AFileSystem.verifyBucketExistsV2()}.
* Value: {@value}
*/
public static final int S3A_BUCKET_PROBE_DEFAULT = 2;
public static final int S3A_BUCKET_PROBE_DEFAULT = 0;
/**
* How long a directory listing in the MS is considered as authoritative.

4
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
View File

@ -1039,9 +1039,9 @@ options are covered in [Testing](./testing.md).
<property>
<name>fs.s3a.bucket.probe</name>
<value>2</value>
<value>0</value>
<description>
The value can be 0, 1 or 2 (default).
The value can be 0 (default), 1 or 2.
When set to 0, bucket existence checks won't be done
during initialization thus making it faster.
Though it should be noted that when the bucket is not available in S3,

1
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
View File

@ -159,6 +159,7 @@ private <E extends Throwable> E expectFSCreateFailure(
return intercept(clazz,
() -> {
fs = S3ATestUtils.createTestFileSystem(conf);
fs.listFiles(new Path("/"), false);
return "expected failure creating FS " + text + " got " + fs;
});
}