big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
60af8793b4
hadoop/hadoop-tools/hadoop-aws/src/main
History
Steve Loughran 511df1e837 HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions 
Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
..
java/org/apache/hadoop/fs HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions 2019-09-05 14:25:15 +01:00
resources/META-INF/services HADOOP-14556. S3A to support Delegation Tokens. 2019-01-14 17:59:27 +00:00
shellprofile.d HADOOP-13345 HS3Guard: Improved Consistency for S3A. 2017-09-01 14:13:41 +01:00