hadoop/hadoop-tools/hadoop-aws
Steve Loughran aaf92fe183
HADOOP-18526. Leak of S3AInstrumentation instances via hadoop Metrics references (#5144)
This has triggered an OOM in a process which was churning through s3a fs
instances; the increased memory footprint of IOStatistics amplified what
must have been a long-standing issue with FS instances being created
and not closed()

*  Makes sure instrumentation is closed when the FS is closed.
*  Uses a weak reference from metrics to instrumentation, so even
   if the FS wasn't closed (see HADOOP-18478), this back reference
   would not cause the S3AInstrumentation reference to be retained.
*  If S3AFileSystem is configured to log at TRACE it will log the
   calling stack of initialize(), so help identify where the
   instance is being created. This should help track down
   the cause of instance leakage.

Contributed by Steve Loughran.
2022-12-14 18:21:03 +00:00
..
dev-support HADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field (#4926) 2022-09-26 18:56:58 +01:00
src HADOOP-18526. Leak of S3AInstrumentation instances via hadoop Metrics references (#5144) 2022-12-14 18:21:03 +00:00
pom.xml HADOOP-18377. hadoop-aws build to add a -prefetch profile to run all tests with prefetching (#4914) 2022-09-20 10:26:13 +01:00