4c55adbb6b
Adds new ClientManager interface/implementation which provides on-demand creation of synchronous and asynchronous s3 clients, s3 transfer manager, and in close() terminates these. S3A FS is modified to * Create a ClientManagerImpl instance and pass down to its S3Store. * Use the same ClientManager interface against S3Store to demand-create the services. * Only create the async client as part of the transfer manager creation, which will take place during the first rename() operation. * Statistics on client creation count and duration are recorded. + Statistics on the time to initialize and shutdown the S3A FS are collected in IOStatistics for reporting. Adds to hadoop common class LazyAtomicReference<T> implements CallableRaisingIOE<T>, Supplier<T> and subclass LazyAutoCloseableReference<T extends AutoCloseable> extends LazyAtomicReference<T> implements AutoCloseable These evaluate the Supplier<T>/CallableRaisingIOE<T> they were constructed with on the first (successful) read of the the value. Any exception raised during this operation will be rethrown, and on future evaluations the same operation retried. These classes implement the Supplier and CallableRaisingIOE interfaces so can actually be used for to implement lazy function evaluation as Haskell and some other functional languages do. LazyAutoCloseableReference is AutoCloseable; its close() method will close the inner reference if it is set This class is used in ClientManagerImpl for the lazy S3 Cliehnt creation and closure. Contributed by Steve Loughran. |
||
---|---|---|
.. | ||
hadoop-annotations | ||
hadoop-auth | ||
hadoop-auth-examples | ||
hadoop-common | ||
hadoop-kms | ||
hadoop-minikdc | ||
hadoop-nfs | ||
hadoop-registry | ||
pom.xml |