big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,382 Commits 2 Branches 2 Tags 264 MiB
e5b76dc99f
Commit Graph

667 Commits

Author SHA1 Message Date
Sean Mackrory
c687a6617d HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota. 2018-06-25 14:59:41 -06:00
Sean Mackrory
55fad6a3de HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota. 2018-06-22 11:36:56 -06:00
Sean Mackrory
b089a06793 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 2018-06-20 16:45:08 -06:00
Chris Douglas
980031bb04 HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs 2018-06-17 11:54:26 -07:00
Aaron Fabbri
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota) 2018-05-29 19:20:22 -07:00
Aaron Fabbri
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota. 2018-05-29 13:38:15 -07:00
Steve Loughran
cc3600aabd HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest. 
Contributed by Steve Loughran.

(cherry picked from commit 170f1040d46f9b1a084e6637def91e9864446acc)
 2018-05-17 19:08:55 +01:00
Steve Loughran
b670837469 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name. 
Contributed by Sean Mackrory.
 2018-05-15 16:13:56 +01:00
Aaron Fabbri
8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota) 2018-05-08 18:58:42 -07:00
Aaron Fabbri
69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota) 2018-05-08 15:29:54 -07:00
Aaron Fabbri
96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri) 2018-05-04 11:34:45 -07:00
Aaron Fabbri
919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota. 2018-04-30 16:02:57 -07:00
Aaron Fabbri
7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota) 2018-04-26 20:41:29 -07:00
Aaron Fabbri
989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota) 2018-04-23 15:26:53 -07:00
Yiqun Lin
1312f9ae4c HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 2018-04-18 16:04:00 +08:00
Aaron Fabbri
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:55:38 -07:00
Aaron Fabbri
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota. 2018-04-05 20:23:17 -07:00
Steve Loughran
5a174f8ac6 HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException. 
Contributed by Gabor Bota.
 2018-04-03 16:31:34 +01:00
Sean Mackrory
7ce6b41509 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-03-20 21:12:20 -07:00
Steve Loughran
1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories. 
Contributed by Steve Loughran.
 2018-03-15 18:05:14 +00:00
Chris Douglas
45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran
dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional. 
Contributed by Steve Loughran.
 2018-03-12 14:01:42 +00:00
Steve Loughran
e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 18:51:10 +00:00
Steve Loughran
8110d6a0d5 HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions. 
Contributed by Aaron Fabbri.
 2018-03-05 14:06:20 +00:00
fang zhenyi
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Steve Loughran
7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. 
Contributed by Steve Loughran.
 2018-02-16 16:37:06 +00:00
Steve Loughran
9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client. 
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb98c2378eed9693a724d4cbbd03c00fd)
 2018-02-15 15:57:10 +00:00
Steve Loughran
b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document. 
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6594c4e519ed39ef36669de2c5cee15)
 2018-02-15 14:57:56 +00:00
Wangda Tan
60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Steve Loughran
1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2018-01-18 13:13:58 +00:00
Steve Loughran
f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits." 
This reverts commit 35ad9b1dd2.
 2018-01-18 12:35:57 +00:00
Aaron Fabbri
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Steve Loughran
a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing 
after OutputCommitter patch.
Contributed by Steve Loughran
 2018-01-15 11:33:32 +00:00
Mukul Kumar Singh
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:45:51 +09:00
Steve Loughran
1a09da7400 HADOOP-15163. Fix S3ACommitter documentation 
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
 2018-01-10 15:37:07 +00:00
Steve Loughran
b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics 
Contributed by Sean Mackrory
 2018-01-09 18:46:52 +00:00
Steve Loughran
c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls. 
Contributed by Steve Loughran
 2017-12-21 14:58:58 +00:00
Steve Loughran
ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance. 
Contributed by Steve Loughran.
 2017-12-21 14:15:53 +00:00
Steve Loughran
1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran
35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2017-12-18 21:19:06 +00:00
Kai Zheng
f86c81d923 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-14 11:14:52 +08:00
Kai Zheng
28792b6b7f Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu" 
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
 2017-12-14 11:05:11 +08:00
Kai Zheng
73b86979d6 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-06 12:01:36 +08:00
Aaron Fabbri
6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory. 2017-12-05 11:06:32 -08:00
Steve Loughran
3150c019ae HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes. 
Contributed by Steve Loughran
 2017-12-05 15:05:41 +00:00
Rohith Sharma K S
3cd75845da MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran. 2017-11-27 22:01:00 +05:30
Steve Loughran
de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
Akira Ajisaka
6903cf096e HADOOP-13514. Upgrade maven surefire plugin to 2.20.1 
Signed-off-by: Allen Wittenauer <aw@apache.org>
 2017-11-19 12:39:37 -08:00
Aaron Fabbri
47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 15:59:38 -07:00
Aaron Fabbri
49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 14:10:48 -07:00
Steve Loughran
5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A. 
Contributed by Aaron Fabbri
 2017-09-08 12:07:02 +01:00
Steve Loughran
83449abb40
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.
 2017-09-07 19:55:56 +01:00
John Zhuge
50506e90a8 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-05 23:26:57 -07:00
Andrew Wang
0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Steve Loughran
621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
 2017-09-01 14:13:41 +01:00
Steve Loughran
7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran

(cherry picked from commit ced547d5f0dbea571cbc472c5f55fe89d5900a6f)
 2017-08-04 11:09:08 +01:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Mingliang Liu
72993b33b7 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran 
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
 2017-06-29 17:07:52 -07:00
Steve Loughran
ee243e5289
HADOOP-14190. Add more on S3 regions to the s3a documentation. 
Contributed by Steve Loughran
 2017-06-28 10:22:13 +01:00
Mingliang Liu
bc4dfe9c9c HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu 2017-06-27 13:19:14 -07:00
Mingliang Liu
f214a9961f HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran 2017-06-15 11:16:16 -07:00
Mingliang Liu
6aeda55bb8 HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 2017-06-05 11:26:56 -07:00
John Zhuge
6c6a7a5962 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 2017-05-28 22:25:00 -07:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Akira Ajisaka
6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu. 2017-05-25 15:36:44 +09:00
Mingliang Liu
6672810eea HADOOP-14432. S3A copyFromLocalFile to be robust, tested. Contributed by Steve Loughran 2017-05-19 11:51:43 -07:00
Steve Loughran
ba70225cf6
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. 
Contributed by Steve Loughran.

(cherry picked from commit 2ac5aab8d725f761a9f9723471a4426f6b5d78c4)
 2017-05-18 15:44:39 +01:00
Akira Ajisaka
0d5c8ed8e0
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor. 2017-05-11 16:37:32 -05:00
Steve Loughran
5f934f8386
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus. 
Contributed by Steve Moist.
 2017-04-24 20:33:19 +01:00
Mingliang Liu
667966c13c HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:13:36 -07:00
Steve Loughran
6b015d00c9
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran
 2017-04-19 10:21:44 +01:00
Mingliang Liu
b053fdc547 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 2017-04-12 14:30:34 -07:00
Chris Nauroth
b8305e6d06 HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu. 2017-04-12 10:02:13 -07:00
Mingliang Liu
5faa949b78 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 2017-04-03 11:07:14 -07:00
Akira Ajisaka
0d053eeb30
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 2017-03-30 18:11:50 +09:00
Andrew Wang
4966a6e26e HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy. 2017-03-29 14:37:21 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Mingliang Liu
2e30aa72e0 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 2017-03-23 11:33:29 -07:00
Steve Loughran
a5a4867f3b
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(). 
Contributed by Yuanbo Liu

(cherry picked from commit 20878d052cebc715c2494a97362fdff08885a77f)
 2017-03-23 12:54:54 +00:00
Steve Loughran
2841666f1f
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9ddacd5fdb740ac1bf9bfeb0ffbbd6c0e)
 2017-03-21 13:15:15 +00:00
Arpit Agarwal
6c399a88e9 HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:23 -07:00
Mingliang Liu
fa67a96d7b HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 14:38:21 -07:00
Mingliang Liu
e6cda5819b Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu" 
This reverts commit b8c69557b7.
 2017-03-14 13:03:42 -07:00
Mingliang Liu
b8c69557b7 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-13 14:15:02 -07:00
Steve Loughran
092ec39fb9
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials. 
Contributed by Yuanbo Liu
 2017-03-10 17:43:22 +00:00
Akira Ajisaka
f597f4c43e
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 2017-03-07 15:14:55 +09:00
Steve Loughran
a97833e0ed
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38bf40373d2855163fa6444821b0f994c)
 2017-03-02 21:44:30 +00:00
Akira Ajisaka
258342e76c HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR. 2017-02-28 13:47:44 +09:00
Steve Loughran
dab00da19f
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.

(cherry picked from commit 29fe5af017b945d8750c074ca39031b5b777eddd)
 2017-02-25 15:35:19 +00:00
Steve Loughran
9c22a91662
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 
(cherry picked from commit ff87ca84418a710c6dc884fe8c70947fcc6489d5)
 2017-02-24 10:41:36 +00:00
Mingliang Liu
a4d4a23785 HADOOP-13321. Deprecate FileSystem APIs that promote inefficient call patterns. Contributed by Chris Nauroth and Mingliang Liu 2017-02-23 12:55:40 -08:00
Mingliang Liu
0013090fb4 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 2017-02-22 13:34:20 -08:00
Steve Loughran
4f4250fbcc HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:43:48 +00:00
Steve Loughran
2158496f6b HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
 2017-02-21 18:28:49 +00:00
Steve Loughran
8035749c26 HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:46 +00:00
Steve Loughran
3a2e30fa9f HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1dc997906390d653dfafb4f0d7e193c4)
 2017-02-18 18:17:11 +00:00
Mingliang Liu
658702efff HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 2017-02-16 16:51:03 -08:00
Mingliang Liu
bdad8b7b97 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 2017-02-16 16:41:31 -08:00
Akira Ajisaka
b9f8491252 HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 2017-02-15 01:45:56 +09:00
Lei Xu
839b690ed5 HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 13:59:03 -08:00
Andrew Wang
5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Steve Loughran
e648b6e138 HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:25:15 +00:00
Mingliang Liu
c6a3923245 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 2016-12-12 14:55:34 -08:00
Mingliang Liu
51211a7d7a HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 2016-12-02 14:01:42 -08:00
Mingliang Liu
2ff84a0040 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 2016-12-02 13:36:04 -08:00
Mingliang Liu
3fd844b99f HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 2016-11-30 13:01:19 -08:00
Mingliang Liu
d60a60be8a HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 2016-11-28 16:30:29 -08:00
Steve Loughran
0de0c32ddd HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:39 +00:00
Mingliang Liu
af791b774b HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:00:35 -08:00
Steve Loughran
a1761a841e HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:54:46 +00:00
Chris Nauroth
1f8490a5ba HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 2016-10-26 08:55:42 -07:00
Chris Nauroth
9cad3e2350 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 2016-10-26 08:27:26 -07:00
Chris Nauroth
309a43925c HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 2016-10-25 09:03:03 -07:00
Chris Nauroth
3372e94030 HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 2016-10-24 21:54:06 -07:00
Chris Nauroth
d8fa1cfa67 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 2016-10-24 21:22:34 -07:00
Anu Engineer
3fbf4cd5da HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:33:58 -07:00
Steve Loughran
9ae270af02 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:59 +01:00
Steve Loughran
6c348c5691 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 21:16:02 +01:00
Chris Nauroth
69620f9559 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 2016-10-07 11:41:19 -07:00
Steve Loughran
ebd4f39a39 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:51:40 +01:00
Steve Loughran
d6be1e75d8 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:02:11 +01:00
Chris Nauroth
88b9444a81 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 2016-10-04 10:36:58 -07:00
Steve Loughran
ee0c722dc8 HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:01:00 +01:00
Chris Nauroth
47f80922dc HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 2016-09-28 16:19:32 -07:00
Mingliang Liu
96142efa2d HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu. 2016-09-17 22:07:46 -07:00
Steve Loughran
4b6d795f28 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:55:32 +01:00
Steve Loughran
7fdfcd8a6c HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:25:23 +01:00
Chris Nauroth
d152557cf7 HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 2016-09-06 09:36:21 -07:00
Chris Douglas
c37346d0e3 HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton. 2016-08-23 14:12:52 -07:00
Chris Nauroth
6f9c346e57 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:18:49 -07:00
Chris Nauroth
763f0497bb HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 2016-08-19 10:48:10 -07:00
Steve Loughran
040c185d62 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:55 +01:00
Chris Nauroth
822d661b8f HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 2016-08-17 14:54:54 -07:00
Chris Nauroth
3808876c73 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 2016-08-16 17:05:52 -07:00
Steve Loughran
86ae218893 HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:50:23 +01:00
Steve Loughran
be9e46b42d HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:45:03 +01:00
Steve Loughran
7052ca8804 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 18:00:08 +01:00
Steve Loughran
37362c2f92 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:51 +01:00
Andrew Wang
da456ffd62 Preparing for 3.0.0-alpha2 development 2016-07-15 19:04:17 -07:00
Jitendra Pandey
c2f9cd584c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:28:53 -07:00
Jitendra Pandey
6e597600f7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 14:58:19 -07:00
Jitendra Pandey
aa42c7a6dd HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:34:03 -07:00
Steve Loughran
96fa0f848b HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu. 2016-06-29 16:04:50 +01:00
Steve Loughran
4ee3543625 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:45:25 +01:00
Chris Nauroth
b2c596cdda HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 2016-06-21 11:28:52 -07:00
Jitendra Pandey
b7c4cf7129 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:25:30 -07:00
Chris Nauroth
127d2c7281 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 2016-06-16 11:18:02 -07:00
Ravi Prakash
4aefe119a0 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 11:13:35 -07:00
Steve Loughran
31ffaf76f2 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 21:00:47 +01:00
Steve Loughran
656c460c0e HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 17:28:49 +01:00
Steve Loughran
a3f78d8fa8 HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:42:36 +02:00
Chris Nauroth
c58a59f708 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:55:33 -07:00
Chris Nauroth
16b1cc7af9 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 2016-06-01 14:49:22 -07:00
Steve Loughran
5870611030 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:18:20 +01:00
Steve Loughran
39ec1515a2 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 16:39:31 +01:00
Steve Loughran
757050ff35 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:15 +01:00
Steve Loughran
c918286b17 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:59 +01:00
Chris Nauroth
93258459fa HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 2016-05-19 22:00:21 -07:00
Steve Loughran
08ea07f1b8 HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:19:29 +01:00
Steve Loughran
dd3a8bed0a HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:47:12 +01:00
Andrew Wang
3c5c57af28 HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1. 2016-05-12 18:27:28 -07:00
Andrew Wang
ca5613af91 Revert "Update project version to 3.0.0-alpha1-SNAPSHOT." 
This reverts commit 6b53802cba.
 2016-05-12 15:32:45 -07:00
Steve Loughran
27c4e90efc HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:24:20 +01:00
Andrew Wang
6b53802cba Update project version to 3.0.0-alpha1-SNAPSHOT. 2016-05-12 11:05:05 -07:00
Steve Loughran
3f2816ab5a HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:35:57 +01:00
Steve Loughran
def2a6d385 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:35 +01:00
Chris Nauroth
27242f211e HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 2016-05-10 15:21:06 -07:00
Steve Loughran
025219b12f HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:37:22 +01:00
Steve Loughran
19f0f9608e HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:25:03 +01:00
Steve Loughran
df18b6e984 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:44:55 +01:00
Steve Loughran
b9e3eff62a HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:25:31 +01:00
Chris Nauroth
843ee8d59d HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 2016-04-08 13:36:58 -07:00
Steve Loughran
ce5b4812f0 HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:10:00 +01:00
Steve Loughran
89c9347511 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:02:04 +01:00
Steve Loughran
0ecdd4cffa HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:40:19 +01:00
Harsh J
256c82fe29 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 2016-04-01 14:18:10 +05:30
Allen Wittenauer
738155063e HADOOP-12857. rework hadoop-tools (aw) 2016-03-23 13:46:38 -07:00
Chris Nauroth
d251e55415 HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 2016-02-29 21:59:52 -08:00
cnauroth
8ab7658025 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 2016-02-18 14:06:38 -08:00
cnauroth
76fab26c5c HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 2016-02-17 12:19:59 -08:00
Steve Loughran
29ae258013 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 15:05:16 +00:00
Ravi Prakash
1acc509b45 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 2016-01-19 13:57:08 -08:00
Lei Xu
126705f67e HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 2016-01-12 12:19:53 -08:00
Ravi Prakash
2d16f40dab HADOOP-12689. S3 filesystem operations stopped working correctly 2016-01-05 23:26:03 -08:00
Akira Ajisaka
3ae82863f0 HADOOP-12553. [JDK8] Fix javadoc error caused by illegal tag. (aajisaka) 2015-11-08 19:17:57 +09:00
Lei Xu
bff7c90a56 HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei) 2015-11-05 18:35:15 -08:00
cnauroth
7fe521b1dd HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu. 2015-09-25 22:33:53 -07:00
Lei Xu
6ab2d19f5c HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu) 2015-08-29 09:59:30 -07:00
Lei Xu
d5403747b5 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 2015-08-04 18:51:52 -07:00
Andrew Wang
2dbc40e608 HADOOP-12059. S3Credentials should support use of CredentialProvider. Contributed by Sean Busbey. 2015-06-05 13:11:01 -07:00
Vinayakumar B
f33efbc827 HADOOP-11922. Misspelling of threshold in log4j.properties for tests in hadoop-tools (Contributed by Gabor Liptak) 2015-05-08 10:22:11 +05:30
Steve Loughran
19262d99eb HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 2015-04-23 21:41:51 +01:00
Tsuyoshi Ozawa
ef9946cd52 HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal tags in hadoop-tools. Contributed by Akira AJISAKA. 2015-03-17 16:09:21 +09:00
Steve Loughran
64443490d7 HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) 2015-03-08 11:22:16 -07:00
Steve Loughran
15b7076ad5 HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel) 2015-03-03 16:18:51 -08:00
Akira Ajisaka
1a625b8158 HADOOP-11480. Typo in hadoop-aws/index.md uses wrong scheme for test.fs.s3.name. Contributed by Ted Yu. 2015-02-24 17:11:46 -08:00
Steve Loughran
709ff99cff HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) 2015-02-21 12:03:03 +00:00
Steve Loughran
aa1c437b6a HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel) 2015-02-20 20:51:24 +00:00
Steve Loughran
00b80958d8 HADOOP-11521. Make connection timeout configurable in s3a. (Thomas Demoor via stevel) 2015-02-17 20:06:27 +00:00
Steve Loughran
78a7e8d3a6 HADOOP-11522. Update S3A Documentation. (Thomas Demoor via stevel) 2015-02-17 18:15:02 +00:00
Steve Loughran
826267f789 HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). 2015-02-17 16:36:44 +00:00
Steve Loughran
17165d3df9 HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) 2015-02-06 21:01:41 +00:00
Steve Loughran
4e7ad4f0a8 HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers. (Ted Yu via stevel) 2015-02-05 12:20:23 +00:00
Harsh J
ffc75d6ebe HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. 2015-02-01 00:17:04 +05:30
Steve Loughran
2908fe4ec5 HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) 2015-01-17 18:40:50 +00:00
Steve Loughran
c6c0f4eb25 HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel) 2015-01-17 18:26:26 +00:00
Steve Loughran
000ca83ea3 HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) 2015-01-16 10:15:51 +00:00
Steve Loughran
27d8395867 HADOOP-11446 S3AOutputStream should use shared thread pool to avoid OutOfMemoryError 2015-01-05 13:00:01 +00:00
Steve Loughran
07bb0b0bbb HADOOP-11412 POMs mention "The Apache Software License" rather than "Apache License". (Herve Boutemy via stevel) 2014-12-16 11:04:36 +00:00
cnauroth
9458cd5bce HADOOP-11394. hadoop-aws documentation missing. Contributed by Chris Nauroth. 2014-12-12 23:29:11 -08:00
Haohui Mai
2e98ad34ce HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack. Contributed by Li Lu. 2014-12-09 20:45:21 -08:00
Aaron T. Myers
6ba52d88ec HADOOP-10714. AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call. Contributed by Juan Yu. 2014-11-05 17:17:04 -08:00
Andrew Wang
4ea77efa3a HADOOP-11140. hadoop-aws only need test-scoped dependency on hadoop-common's tests jar. Contributed by Juan Yu. 2014-09-25 21:56:26 -07:00
Colin Patrick Mccabe
0ac760a58d HADOOP-11091. Eliminate old configuration parameter names from s3a (dsw via cmccabe) 2014-09-15 16:49:15 -07:00
Aaron T. Myers
24d920b80e HADOOP-10400. Incorporate new S3A FileSystem implementation. Contributed by Jordan Mendelson and Dave Wang. 2014-09-15 08:27:07 -07:00
Colin Patrick Mccabe
5ec7fcd9dd HADOOP-11074. Move s3-related FS connector code to hadoop-aws. (David S. Wang via Colin Patrick McCabe) 2014-09-10 16:14:53 -07:00
Steve Loughran
59384dfb71 HADOOP-10373 create tools/hadoop-amazon for aws/EMR support 2014-09-02 20:11:13 +01:00