hadoop/hadoop-tools
Steve Loughran 20eec95867
HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936)
Contributed by Steve Loughran.

This strips out all the -p preservation options which have already been
processed when uploading a file before deciding whether or not to query
the far end for the status of the (existing/uploaded) file to see if any
other attributes need changing.

This will avoid 404 caching-related issues in S3, wherein a newly created
file can have a 404 entry in the S3 load balancer's cache from the
probes for the file's existence prior to the upload.

It partially addresses a regression caused by HADOOP-8143,
"Change distcp to have -pb on by default" that causes a resurfacing
of HADOOP-13145, "In DistCp, prevent unnecessary getFileStatus call when
not preserving metadata"
2020-04-07 17:55:55 +01:00
..
hadoop-aliyun Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-archive-logs Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-archives Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-aws Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-azure HADOOP-16910 : ABFS Streams to update FileSystem.Statistics counters on IO. (#1918). Contributed by Mehakmeet Singh. 2020-03-31 14:49:09 +02:00
hadoop-azure-datalake Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-datajoin Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-distcp HADOOP-16932. distcp copy calls getFileStatus() needlessly and can fail against S3 (#1936) 2020-04-07 17:55:55 +01:00
hadoop-dynamometer Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-extras Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-fs2img Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-gridmix Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-kafka Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-openstack Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-rumen Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-sls Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-streaming Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-tools-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
pom.xml Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30