hadoop/hadoop-tools/hadoop-aws
Steve Loughran 7651afd3db
HADOOP-19057. S3A: Landsat bucket used in tests no longer accessible (#6515)
The AWS landsat data previously used in some S3A tests is no
longer accessible

This PR moves to the new external file
s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz

* Large enough file for scale tests
* Bucket supports anonymous access
* Ends in .gz to keep codec tests happy
* No spaces in path to keep bucket-info happy

Test Code Changes
* Leaves the test key name alone: fs.s3a.scale.test.csvfile
* Rename all methods and fields move remove "csv" from their names and
  move to "external file" we no longer require it to be CSV.
* Path definition and helper methods have been moved to PublicDatasetTestUtils
* Improve error reporting in ITestS3AInputStreamPerformance if the file
  is too short
  
With S3 Select removed, there is no need for the file to be
a CSV file; there is a test which tries to unzip it; other
tests have a minimum file size.

Consult the JIRA for the settings to add to auth-keys.xml
to switch earlier builds to this same file.

Contributed by Steve Loughran
2024-02-13 10:46:36 +00:00
..
dev-support HADOOP-18073. S3A: Upgrade AWS SDK to V2 (#5995) 2023-09-11 14:30:25 +01:00
src HADOOP-19057. S3A: Landsat bucket used in tests no longer accessible (#6515) 2024-02-13 10:46:36 +00:00
pom.xml HADOOP-18830. Cut S3 Select (#6144) 2024-01-30 16:12:27 +00:00