hadoop/hadoop-common-project
Steve Loughran e199da3fae
HADOOP-17833. Improve Magic Committer performance (#3289)
Speed up the magic committer with key changes being

* Writes under __magic always retain directory markers

* File creation under __magic skips all overwrite checks,
  including the LIST call intended to stop files being
	created over dirs.
* mkdirs under __magic probes the path for existence
  but does not look any further.  	

Extra parallelism in task and job commit directory scanning
Use of createFile and openFile with parameters which all for
HEAD checks to be skipped.

The committer can write the summary _SUCCESS file to the path
`fs.s3a.committer.summary.report.directory`, which can be in a
different file system/bucket if desired, using the job id as
the filename. 

Also: HADOOP-15460. S3A FS to add `fs.s3a.create.performance`

Application code can set the createFile() option
fs.s3a.create.performance to true to disable the same
safety checks when writing under magic directories.
Use with care.

The createFile option prefix `fs.s3a.create.header.`
can be used to add custom headers to S3 objects when
created.


Contributed by Steve Loughran.
2022-06-17 19:11:35 +01:00
..
hadoop-annotations HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 2021-10-06 02:17:05 +09:00
hadoop-auth HADOOP-18120. Hadoop auth does not handle HTTP Headers in a case-insensitive way. Contributed by Janos Makai. 2022-05-20 10:43:19 +02:00
hadoop-auth-examples HADOOP-18120. Hadoop auth does not handle HTTP Headers in a case-insensitive way. Contributed by Janos Makai. 2022-05-20 10:43:19 +02:00
hadoop-common HADOOP-17833. Improve Magic Committer performance (#3289) 2022-06-17 19:11:35 +01:00
hadoop-kms HADOOP-18289. Remove WhiteBox in hadoop-kms module. (#4433) 2022-06-17 09:13:16 +08:00
hadoop-minikdc HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-nfs HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 2021-11-15 22:57:24 +09:00
hadoop-registry HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 2021-12-10 01:36:31 +09:00
pom.xml Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30