hadoop/hadoop-tools
Steve Loughran 7999db55da
HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151)
If a file is opened for reading through the S3A connector
is not closed, then when garbage collection takes place

* An error message is reported at WARN, including the file name.
* A stack trace of where the stream was created is reported
  at INFO.
* A best-effort attempt is made to release any active HTTPS
  connection.
* The filesystem IOStatistic stream_leaks is incremented.

The intent is to make it easier to identify where streams
are being opened and not closed -as these consume resources
including often HTTPS connections from the connection pool
of limited size.

It MUST NOT be relied on as a way to clean up open
files/streams automatically; some of the normal actions of
the close() method are omitted.

Instead: view the warning messages and IOStatistics as a
sign of a problem, the stack trace as a way of identifying
what application code/library needs to be investigated.

Contributed by Steve Loughran
2024-11-14 17:02:25 +00:00
..
hadoop-aliyun HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 2024-09-24 16:50:22 +01:00
hadoop-archive-logs HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-archives HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-aws HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151) 2024-11-14 17:02:25 +00:00
hadoop-azure HADOOP-18960: [ABFS] Making Contract tests run in sequential and Other Test Fixes (#7104) 2024-11-05 16:12:03 -06:00
hadoop-azure-datalake Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-benchmark Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-compat-bench HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-datajoin HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-distcp HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-dynamometer HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-extras HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-federation-balance HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-fs2img HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-gridmix HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-kafka HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-openstack Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-pipes Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-resourceestimator Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-rumen HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128) 2024-11-11 15:46:36 +00:00
hadoop-sls HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-streaming HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 2024-11-05 17:35:53 +00:00
hadoop-tools-dist Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
pom.xml HADOOP-19085. Compatibility Benchmark over HCFS Implementations 2024-03-17 16:48:29 +08:00