hadoop/hadoop-common-project
Steve Loughran 7999db55da
HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151)
If a file is opened for reading through the S3A connector
is not closed, then when garbage collection takes place

* An error message is reported at WARN, including the file name.
* A stack trace of where the stream was created is reported
  at INFO.
* A best-effort attempt is made to release any active HTTPS
  connection.
* The filesystem IOStatistic stream_leaks is incremented.

The intent is to make it easier to identify where streams
are being opened and not closed -as these consume resources
including often HTTPS connections from the connection pool
of limited size.

It MUST NOT be relied on as a way to clean up open
files/streams automatically; some of the normal actions of
the close() method are omitted.

Instead: view the warning messages and IOStatistics as a
sign of a problem, the stack trace as a way of identifying
what application code/library needs to be investigated.

Contributed by Steve Loughran
2024-11-14 17:02:25 +00:00
..
hadoop-annotations
hadoop-auth
hadoop-auth-examples
hadoop-common HADOOP-19330. S3A: Add LeakReporter; use in S3AInputStream (#7151) 2024-11-14 17:02:25 +00:00
hadoop-kms
hadoop-minikdc
hadoop-nfs
hadoop-registry
pom.xml