hadoop/hadoop-tools/hadoop-azure
Mehakmeet Singh acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446)
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
2021-09-21 12:48:06 +01:00
..
dev-support Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml (#3223) 2021-07-25 13:10:27 +05:30
src HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 2021-09-21 12:48:06 +01:00
.gitignore HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results 2020-12-16 10:34:59 -08:00
pom.xml HADOOP-17156. ABFS: Release the byte buffers held by input streams in close() (#3285) 2021-09-07 15:13:36 +05:30