b6dc00f481
Contributed by Amir Shenavandeh. This avoids overwrite consistency issues with S3 and other stores -though given S3's copy operation is O(data), you are still best of using -direct when distcp-ing to it. Change-Id: I8dc9f048ad0cc57ff01543b849da1ce4eaadf8c3
DistCp (distributed copy) is a tool used for large inter/intra-cluster copying. It uses Map/Reduce to effect its distribution, error handling and recovery, and reporting. It expands a list of files and directories into input to map tasks, each of which will copy a partition of the files specified in the source list.