hadoop/hadoop-common-project/hadoop-common/src/main
Steve Loughran 1ef34d0819
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.

Change-Id: I57161b026f28349e339dc8b9d74f6567a62ce196
2020-11-25 14:55:29 +00:00
..
arm-java/org/apache/hadoop/ipc/protobuf HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
bin HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:50:10 +09:00
conf HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:50:10 +09:00
java/org/apache/hadoop HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 2020-11-25 14:55:29 +00:00
native HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 2020-11-10 11:41:21 -08:00
proto HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
resources HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 2020-07-13 13:32:04 +01:00
webapps/static
winutils
xsl