hadoop/hadoop-tools
Steve Loughran 4249c04d45
HADOOP-16798. S3A Committer thread pool shutdown problems. (#1963)
Contributed by Steve Loughran.

Fixes a condition which can cause job commit to fail if a task was
aborted < 60s before the job commit commenced: the task abort
will shut down the thread pool with a hard exit after 60s; the
job commit POST requests would be scheduled through the same pool,
so be interrupted and fail. At present the access is synchronized,
but presumably the executor shutdown code is calling wait() and releasing
locks.

Task abort is triggered from the AM when task attempts succeed but
there are still active speculative task attempts running. Thus it
only surfaces when speculation is enabled and the final tasks are
speculating, which, given they are the stragglers, is not unheard of.

Note: this problem has never been seen in production; it has surfaced
in the hadoop-aws tests on a heavily overloaded desktop
2020-06-30 10:44:51 +01:00
..
hadoop-aliyun Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-archive-logs Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-archives Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-aws HADOOP-16798. S3A Committer thread pool shutdown problems. (#1963) 2020-06-30 10:44:51 +01:00
hadoop-azure HADOOP-17089: WASB: Update azure-storage-java SDK 2020-06-25 02:32:42 +00:00
hadoop-azure-datalake Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-datajoin Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-distcp Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 2020-05-14 19:04:36 +01:00
hadoop-dynamometer Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-extras Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-federation-balance HDFS-15428. Javadocs fails for hadoop-federation-balance. Contributed by Xieming Li. 2020-06-22 19:43:19 +09:00
hadoop-fs2img Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-gridmix HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest 2020-04-30 14:15:28 -07:00
hadoop-kafka Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-openstack Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-rumen Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-sls YARN-9301. Too many InvalidStateTransitionException with SLS. Contributed by Bilwa S T. 2020-05-12 08:24:34 -07:00
hadoop-streaming Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-tools-dist HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
pom.xml HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00