hadoop/hadoop-tools/hadoop-aws
Steve Loughran 511df1e837 HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions
Contributed by Steve Loughran.

This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.

For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.

Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
..
dev-support HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 2019-02-05 11:51:02 +00:00
src HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions 2019-09-05 14:25:15 +01:00
pom.xml HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3. 2019-07-12 13:02:25 +01:00