hadoop/hadoop-tools/hadoop-aws
Steve Loughran 9960c01a25
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoke.

This avoids directory tombstones being added above files/child directories,
which stop the treewalk and delete phase from working.

Also:

* Callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* New statistic to measure #of objects deleted, alongside request count.
* Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
  to clarify behavior.
* Test enhancements to replicate the failure and verify the fix

Contributed by Steve Loughran
2020-09-10 17:03:52 +01:00
..
dev-support HADOOP-16697. Tune/audit S3A authoritative mode. 2020-01-10 11:11:56 +00:00
src HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280) 2020-09-10 17:03:52 +01:00
pom.xml HADOOP-13230. S3A to optionally retain directory markers. 2020-08-15 12:51:08 +01:00