hadoop/hadoop-tools/hadoop-aws/src/main
Steve Loughran f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
..
java/org/apache/hadoop/fs HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 2019-02-05 11:51:02 +00:00
resources/META-INF/services HADOOP-14556. S3A to support Delegation Tokens. 2019-01-14 17:59:27 +00:00
shellprofile.d HADOOP-13345 HS3Guard: Improved Consistency for S3A. 2017-09-01 14:13:41 +01:00