hadoop/hadoop-tools/hadoop-azure
Mehakmeet Singh 823f5ee0d4
HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state (#4331)
ABFS rename fails intermittently when the Storage-blob tracking
metadata is in an incomplete state. This surfaces as the error code
404 and an error message of "RenameDestinationParentPathNotFound"

To mitigate this issue, when a request fails with this response.
the ABFS client issues a HEAD call on the source file
and then retries the rename operation again

ABFS filesystem statistics track when this occurs with new counters
  rename_recovery
  metadata_incomplete_rename_failures
  rename_path_attempts

This is very rare occurrence and appears to be triggered under certain
heavy load conditions, just as with HADOOP-18163.

Contributed by Mehakmeet Singh.
2022-06-27 19:06:59 +01:00
..
dev-support Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml (#3223) 2021-07-25 13:10:27 +05:30
src HADOOP-18242. ABFS Rename Failure when tracking metadata is in an incomplete state (#4331) 2022-06-27 19:06:59 +01:00
.gitignore HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results 2020-12-16 10:34:59 -08:00
pom.xml HADOOP-18163. hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 2022-03-17 11:24:51 +00:00