hadoop/hadoop-tools/hadoop-azure
Steve Loughran 936e9e15d0
MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519)
This modifies the manifest committer so that the list of files
to rename is passed between stages as a file of
writeable entries on the local filesystem.

The map of directories to create is still passed in memory;
this map is built across all tasks, so even if many tasks
created files, if they all write into the same set of directories
the memory needed is O(directories) with the
task count not a factor.

The _SUCCESS file reports on heap size through gauges.
This should give a warning if there are problems.

Contributed by Steve Loughran
2023-06-12 13:43:43 +01:00
..
dev-support HADOOP-17770 WASB : Support disabling buffered reads in positional reads (#3233) 2021-10-22 11:45:42 +05:30
src MAPREDUCE-7435. Manifest Committer OOM on abfs (#5519) 2023-06-12 13:43:43 +01:00
.gitignore HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results (#2597) 2021-03-10 18:25:41 +00:00
pom.xml HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429) 2023-02-28 14:05:13 +00:00