hadoop/hadoop-tools/hadoop-aws
Steve Loughran 7999db55da
HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151)
If a file is opened for reading through the S3A connector
is not closed, then when garbage collection takes place

* An error message is reported at WARN, including the file name.
* A stack trace of where the stream was created is reported
  at INFO.
* A best-effort attempt is made to release any active HTTPS
  connection.
* The filesystem IOStatistic stream_leaks is incremented.

The intent is to make it easier to identify where streams
are being opened and not closed -as these consume resources
including often HTTPS connections from the connection pool
of limited size.

It MUST NOT be relied on as a way to clean up open
files/streams automatically; some of the normal actions of
the close() method are omitted.

Instead: view the warning messages and IOStatistics as a
sign of a problem, the stack trace as a way of identifying
what application code/library needs to be investigated.

Contributed by Steve Loughran
2024-11-14 17:02:25 +00:00
..
dev-support HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) 2024-07-05 16:38:37 +01:00
src HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151) 2024-11-14 17:02:25 +00:00
pom.xml HADOOP-18708: S3A: Support S3 Client Side Encryption(CSE) (#6884) 2024-11-14 13:39:56 +00:00