hadoop/hadoop-tools
Steve Loughran ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.
2020-11-25 14:31:02 +00:00
..
hadoop-aliyun HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-archive-logs HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-archives HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-aws HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 2020-11-25 14:31:02 +00:00
hadoop-azure HADOOP-17311. ABFS: Logs should redact SAS signature (#2422) 2020-11-25 14:22:10 +00:00
hadoop-azure-datalake HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-datajoin Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-distcp HDFS-15608.Reset the DistCp#CLEANUP variable definition. (#2351). Contributed by JiangHua Zhu. 2020-11-10 13:02:29 -08:00
hadoop-dynamometer HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 2020-11-19 14:37:14 -06:00
hadoop-extras Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-federation-balance HDFS-15640. Add diff threshold to FedBalance. Contributed by Jinglun. 2020-10-27 10:41:10 +08:00
hadoop-fs2img HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-gridmix HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-kafka HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-openstack Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-rumen Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-sls YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 16:55:06 +00:00
hadoop-streaming Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-tools-dist HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
pom.xml HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00