Go to file
Steve Loughran e0cd0a82e0
HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3)
S3A input stream support for the few fs.option.openfile settings.
As well as supporting the read policy option and values,
if the file length is declared in fs.option.openfile.length
then no HEAD request will be issued when opening a file.
This can cut a few tens of milliseconds off the operation.

The patch adds a new openfile parameter/FS configuration option
fs.s3a.input.async.drain.threshold (default: 16000).
It declares the number of bytes remaining in the http input stream
above which any operation to read and discard the rest of the stream,
"draining", is executed asynchronously.
This asynchronous draining offers some performance benefit on seek-heavy
file IO.

Contributed by Steve Loughran.

Change-Id: I9b0626bbe635e9fd97ac0f463f5e7167e0111e39
2022-04-24 17:33:05 +01:00
.github HADOOP-17799. Improve the GitHub pull request template (#3277) 2021-08-14 21:16:15 +09:00
dev-support HADOOP-18202. create-release fails fatal: unsafe repository (#4188) 2022-04-18 19:24:35 +01:00
hadoop-assemblies HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
hadoop-build-tools HADOOP-17968 Migrate checkstyle module illegalimport to maven enforcer banned-illegal-imports (#3584) 2021-10-28 15:57:15 +09:00
hadoop-client-modules HADOOP-18160 Avoid shading wildfly.openssl runtime dependency (#4074) 2022-03-22 17:14:29 +00:00
hadoop-cloud-storage-project HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-common-project HADOOP-16202. Enhanced openFile(): hadoop-common changes. (#2584/1) 2022-04-24 17:33:04 +01:00
hadoop-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-hdfs-project Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232) 2022-04-24 17:18:31 +08:00
hadoop-mapreduce-project HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 2022-04-24 17:33:05 +01:00
hadoop-maven-plugins HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-minicluster HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-project HADOOP-18196. Remove replace-guava from replacer plugin (#4152) 2022-04-15 23:24:02 +09:00
hadoop-project-dist Make upstream aware of 3.3.2 release 2022-03-02 19:14:50 -08:00
hadoop-tools HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3) 2022-04-24 17:33:05 +01:00
hadoop-yarn-project HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 2022-04-24 17:33:05 +01:00
licenses HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
licenses-binary HADOOP-15993. Upgrade Kafka to 2.4.0 in hadoop-kafka module. (#1796) 2020-01-09 16:24:58 +09:00
.asf.yaml HADOOP-17234. Add .asf.yaml to allow Github to Jira integration. (#2253). Contributed by Ayush Saxena. 2020-08-28 17:22:46 +05:30
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 2020-09-01 10:44:55 +09:00
BUILDING.txt Update BUILDING.txt (#3811) 2021-12-22 13:08:14 +08:00
LICENSE-binary HADOOP-18178. Upgrade jackson to 2.13.2 and jackson-databind to 2.13.2.2 (#4111) 2022-04-07 16:19:36 +09:00
LICENSE.txt HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 2022-01-12 11:40:32 +08:00
NOTICE-binary HADOOP-18068. upgrade AWS SDK to 1.12.132 (#3864) 2022-01-18 10:31:28 +00:00
NOTICE.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
pom.xml HADOOP-17551. Upgrade maven-site-plugin to 3.11.0 (#4196) 2022-04-21 22:16:12 +09:00
README.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
start-build-env.sh HADOOP-18052. Support Apple Silicon in start-build-env.sh (#3817) 2021-12-23 18:13:18 +09:00

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   https://cwiki.apache.org/confluence/display/HADOOP/