hadoop/hadoop-tools/hadoop-azure
Steve Loughran ee466d4b40
HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240)
This patch cuts down the size of directory trees used for
distcp contract tests against object stores, so making
them much faster against distant/slow stores.

On abfs, the test only runs with -Dscale (as was the case for s3a already),
and has the larger scale test timeout.

After every test case, the FileSystem IOStatistics are logged,
to provide information about what IO is taking place and
what it's performance is.

There are some test cases which upload files of 1+ MiB; you can
increase the size of the upload in the option
"scale.test.distcp.file.size.kb" 
Set it to zero and the large file tests are skipped.

Contributed by Steve Loughran.
2021-08-02 11:36:43 +01:00
..
dev-support Addendum HADOOP-17770 WASB : Support disabling buffered reads in positional reads - Added the invalid SpotBugs warning to findbugs-exclude.xml (#3223) 2021-07-25 13:10:27 +05:30
src HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 2021-08-02 11:36:43 +01:00
.gitignore HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results 2020-12-16 10:34:59 -08:00
pom.xml HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00