hadoop/hadoop-common-project
Steve Loughran 4c55adbb6b
HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892)
Adds new ClientManager interface/implementation which provides on-demand
creation of synchronous and asynchronous s3 clients, s3 transfer manager,
and in close() terminates these.

S3A FS is modified to
* Create a ClientManagerImpl instance and pass down to its S3Store.
* Use the same ClientManager interface against S3Store to demand-create
  the services.
* Only create the async client as part of the transfer manager creation,
  which will take place during the first rename() operation.
* Statistics on client creation count and duration are recorded.
+ Statistics on the time to initialize and shutdown the S3A FS are collected
  in IOStatistics for reporting.

Adds to hadoop common class
  LazyAtomicReference<T> implements CallableRaisingIOE<T>, Supplier<T>
and subclass
  LazyAutoCloseableReference<T extends AutoCloseable>
    extends LazyAtomicReference<T> implements AutoCloseable

These evaluate the Supplier<T>/CallableRaisingIOE<T> they were
constructed with on the first (successful) read of the the value.
Any exception raised during this operation will be rethrown, and on future
evaluations the same operation retried.

These classes implement the Supplier and CallableRaisingIOE
interfaces so can actually be used for to implement lazy function evaluation
as Haskell and some other functional languages do.

LazyAutoCloseableReference is AutoCloseable; its close() method will
close the inner reference if it is set

This class is used in ClientManagerImpl for the lazy S3 Cliehnt creation
and closure.

Contributed by Steve Loughran.
2024-07-05 16:38:37 +01:00
..
hadoop-annotations Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
hadoop-auth HADOOP-19160. hadoop-auth should not depend on kerb-simplekdc (#6788) 2024-05-03 12:57:26 +02:00
hadoop-auth-examples HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 2024-02-13 16:33:51 +00:00
hadoop-common HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892) 2024-07-05 16:38:37 +01:00
hadoop-kms HDFS-13603: Do not propagate ExecutionException while initializing EDEK queues for keys. (#6860) 2024-06-03 09:10:06 -07:00
hadoop-minikdc HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 2024-02-13 16:33:51 +00:00
hadoop-nfs HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 2024-02-13 16:33:51 +00:00
hadoop-registry Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00
pom.xml Preparing for 3.5.0 development (#6411) 2024-01-19 15:05:22 +08:00