big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HADOOP-18407. Improve readVectored() api spec (#4760)

Browse Source 
part of HADOOP-18103.

Contributed By: Mukund Thakur
This commit is contained in:
Mukund Thakur 2022-08-22 23:19:29 +05:30 committed by GitHub
parent a9e5fb3313
commit 231e095802
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 17 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PositionedReadable.java
View File

@ -114,6 +114,16 @@ default int maxReadSizeForVectorReads() {
* As a result of the call, each range will have FileRange.setData(CompletableFuture) * As a result of the call, each range will have FileRange.setData(CompletableFuture)
* called with a future that when complete will have a ByteBuffer with the * called with a future that when complete will have a ByteBuffer with the
* data from the file's range. * data from the file's range.
* <p>
* The position returned by getPos() after readVectored() is undefined.
* </p>
* <p>
* If a file is changed while the readVectored() operation is in progress, the output is
* undefined. Some ranges may have old data, some may have new and some may have both.
* </p>
* <p>
* While a readVectored() operation is in progress, normal read api calls may block.
* </p>
* @param ranges the byte ranges to read * @param ranges the byte ranges to read
* @param allocate the function to allocate ByteBuffer * @param allocate the function to allocate ByteBuffer
* @throws IOException any IOE. * @throws IOException any IOE.

7
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdatainputstream.md
View File

@ -454,6 +454,13 @@ Also, clients are encouraged to use `WeakReferencedElasticByteBufferPool` for
allocating buffers such that even direct buffers are garbage collected when allocating buffers such that even direct buffers are garbage collected when
they are no longer referenced. they are no longer referenced.
The position returned by `getPos()` after `readVectored()` is undefined.
If a file is changed while the `readVectored()` operation is in progress, the output is
undefined. Some ranges may have old data, some may have new, and some may have both.
While a `readVectored()` operation is in progress, normal read api calls may block.
Note: Don't use direct buffers for reading from ChecksumFileSystem as that may Note: Don't use direct buffers for reading from ChecksumFileSystem as that may
lead to memory fragmentation explained in HADOOP-18296. lead to memory fragmentation explained in HADOOP-18296.