Go to file
Steve Loughran ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.
2020-11-25 14:31:02 +00:00
.github HADOOP-15184. Add GitHub pull request template. (#1419) 2019-09-11 11:10:11 +09:00
dev-support HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. (#2433) 2020-11-10 13:58:27 +09:00
hadoop-assemblies HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
hadoop-build-tools HADOOP-17319. Update the checkstyle config to ban some guava functions. (#2400) 2020-10-23 04:28:17 +09:00
hadoop-client-modules HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade (#2458) 2020-11-11 11:37:23 -06:00
hadoop-cloud-storage-project HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-common-project HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 2020-11-25 14:31:02 +00:00
hadoop-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-hdfs-project HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 2020-11-23 11:26:52 +08:00
hadoop-mapreduce-project MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan 2020-11-24 17:02:35 +01:00
hadoop-maven-plugins HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-minicluster HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji 2020-05-06 02:06:38 +05:30
hadoop-project HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468) 2020-11-23 14:08:12 +00:00
hadoop-project-dist HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 2020-10-06 17:07:54 +01:00
hadoop-tools HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 2020-11-25 14:31:02 +00:00
hadoop-yarn-project YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485) 2020-11-25 13:00:09 +09:00
licenses HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
licenses-binary HADOOP-15993. Upgrade Kafka to 2.4.0 in hadoop-kafka module. (#1796) 2020-01-09 16:24:58 +09:00
.asf.yaml HADOOP-17234. Add .asf.yaml to allow Github to Jira integration. (#2253). Contributed by Ayush Saxena. 2020-08-28 17:22:46 +05:30
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 2020-09-01 10:44:55 +09:00
BUILDING.txt HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 2020-10-06 17:07:54 +01:00
Jenkinsfile HADOOP-17297. Use Yetus before YETUS-994 to enable adding comments to GitHub (#2359) 2020-10-05 19:16:37 +09:00
LICENSE-binary HADOOP-17343. Upgrade AWS SDK to 1.11.901 (#2468) 2020-11-23 14:08:12 +00:00
LICENSE.txt HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 2020-09-24 21:15:55 +09:00
NOTICE-binary HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
NOTICE.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
pom.xml upate the hadoop.version property in the root pom.xml and hadoop.assemblies.version in hadoop-project/pom.xml (see HADOOP-15369) 2020-03-29 23:44:20 +05:30
README.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
start-build-env.sh HADOOP-17298. Backslash in username causes build failure in the environment started by start-build-env.sh. (#2367) 2020-10-20 10:09:03 +09:00

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   https://cwiki.apache.org/confluence/display/HADOOP/