From 0d057e27c3cab15c4af270950abda7fa73e56ab4 Mon Sep 17 00:00:00 2001 From: Steve Loughran Date: Tue, 27 Jun 2023 13:55:20 +0100 Subject: [PATCH] MAPREDUCE-7432. Make manifest committer default on abfs and gcs stores (#5378) By default, the mapreduce manifest committer is used for jobs working with abfs and gcs. Hadoop mapreduce will pick this up automatically; for Spark it is a bit complicated: read the docs to see the steps required. --- .../src/main/resources/mapred-default.xml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml index 7aadd52109..2921b31caf 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml @@ -2235,23 +2235,23 @@ - mapreduce.outputcommitter.factory.scheme.abfs org.apache.hadoop.fs.azurebfs.commit.AzureManifestCommitterFactory - The default committer factory for ABFS is for the manifest committer with - abfs-specific tuning. + The default committer factory for ABFS is the manifest committer with + abfs-specific recovery. + mapreduce.outputcommitter.factory.scheme.gs org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory - The default committer factory for google cloud storage is for the manifest committer. + The default committer factory for google cloud storage is the manifest committer. ---> +