hadoop/hadoop-tools
Steve Loughran e4331a73c9
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936)
Contributed by Steve Loughran.

This strips out all the -p preservation options which have already been
processed when uploading a file before deciding whether or not to query
the far end for the status of the (existing/uploaded) file to see if any
other attributes need changing.

This will avoid 404 caching-related issues in S3, wherein a newly created
file can have a 404 entry in the S3 load balancer's cache from the
probes for the file's existence prior to the upload.

It partially addresses a regression caused by HADOOP-8143,
"Change distcp to have -pb on by default" that causes a resurfacing
of HADOOP-13145, "In DistCp, prevent unnecessary getFileStatus call when
not preserving metadata"

Change-Id: Ibc25d19e92548e6165eb8397157ebf89446333f7
2020-04-09 18:23:47 +01:00
..
hadoop-aliyun HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 2020-03-08 21:01:34 -07:00
hadoop-archive-logs HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-archives HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements 2019-10-07 16:38:08 +09:00
hadoop-aws HADOOP-16939 fs.s3a.authoritative.path should support multiple FS URIs (#1914) 2020-03-26 12:59:11 -06:00
hadoop-azure Revert "HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob" 2020-03-26 15:24:37 +00:00
hadoop-azure-datalake HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden 2020-03-09 14:44:28 +00:00
hadoop-datajoin HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-distcp HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936) 2020-04-09 18:23:47 +01:00
hadoop-dynamometer YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
hadoop-extras HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-fs2img HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
hadoop-gridmix HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-kafka HADOOP-16512. [hadoop-tools] Fix order of actual and expected expression in assert statements 2019-10-07 16:38:08 +09:00
hadoop-openstack HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden 2020-03-09 14:44:28 +00:00
hadoop-pipes HADOOP-16739. Fix native build failure of hadoop-pipes on CentOS 8. 2020-02-10 13:13:11 +09:00
hadoop-resourceestimator HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-rumen HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
hadoop-sls YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
hadoop-streaming YARN-8767. TestStreamingStatus fails. Contributed by Andras Bokor 2020-02-25 21:48:16 +01:00
hadoop-tools-dist HADOOP-16331. Fix ASF License check in pom.xml 2019-05-29 17:25:13 +09:00
pom.xml HDFS-12345 Add Dynamometer to hadoop-tools, a tool for scale testing the HDFS NameNode with real metadata and workloads. Contributed by Erik Krogen. 2019-06-25 08:07:39 -07:00