hadoop/hadoop-common-project/hadoop-common
Steve Loughran 0c82eb0324
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoke.

This avoids directory tombstones being added above files/child directories,
which stop the treewalk and delete phase from working.

Also:

* Callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* New statistic to measure #of objects deleted, alongside request count.
* Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
  to clarify behavior.
* Test enhancements to replicate the failure and verify the fix

Contributed by Steve Loughran

Change-Id: I0e6ea2c35e487267033b1664228c8837279a35c7
2020-09-10 17:29:33 +01:00
..
dev-support HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
src HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280) 2020-09-10 17:29:33 +01:00
HadoopCommon.cmake HADOOP-17196. Fix C/C++ standard warnings (#2208) 2020-08-11 16:35:41 +09:00
HadoopJNI.cmake HDFS-13822. speedup libhdfs++ build (enable parallel build). Contributed by Allen Wittenauer and Pradeep Ambati 2018-08-17 12:25:36 -05:00
pom.xml HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30