big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-18246. Reduce lower limit on fs.s3a.prefetch.block.size to 1 byte. (#5120)

Browse Source 
The minimum value of fs.s3a.prefetch.block.size is now 1

Contributed by Ankit Saurabh
This commit is contained in:
Ankit Saurabh 2023-02-02 18:45:21 +00:00 committed by GitHub
parent ad0cff2f97
commit 22f6d55b71
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 6 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
View File

@ -531,8 +531,7 @@ public void initialize(URI name, Configuration originalConf)
this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT); this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
long prefetchBlockSizeLong = long prefetchBlockSizeLong =
longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, 1);
PREFETCH_BLOCK_DEFAULT_SIZE);
if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) { if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) {
throw new IOException("S3A prefatch block size exceeds int limit"); throw new IOException("S3A prefatch block size exceeds int limit");
} }

1
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
View File

@ -1108,6 +1108,7 @@ options are covered in [Testing](./testing.md).
<value>8MB</value> <value>8MB</value>
<description> <description>
The size of a single prefetched block of data. The size of a single prefetched block of data.
Decreasing this will increase the number of prefetches required, and may negatively impact performance.
</description> </description>
</property> </property>

4
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md
View File

@ -43,6 +43,10 @@ Multiple blocks may be read in parallel.
|`fs.s3a.prefetch.block.size` |Size of a block |`8M` | |`fs.s3a.prefetch.block.size` |Size of a block |`8M` |
|`fs.s3a.prefetch.block.count` |Number of blocks to prefetch |`8` | |`fs.s3a.prefetch.block.count` |Number of blocks to prefetch |`8` |
The default size of a block is 8MB, and the minimum allowed block size is 1 byte.
Decreasing block size will increase the number of blocks to be read for a file.
A smaller block size may negatively impact performance as the number of prefetches required will increase.
### Key Components ### Key Components
`S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of `S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of