big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
21,478 Commits 2 Branches 2 Tags 264 MiB
c0427c84dd
Commit Graph

1121 Commits

Author SHA1 Message Date
Vishwajeet Dusane
784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress. 
Contributed by Vishwajeet Dusane.
 2018-12-20 18:00:05 +00:00
Da Zhou
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.
 2018-12-17 11:15:20 +00:00
Da Zhou
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb6889e51d4d4c914eb6159e40845e9f)
 2018-12-17 11:10:22 +00:00
Da Zhou
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2018-12-17 11:04:40 +00:00
Sean Mackrory
82b798581d HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota. 2018-12-12 09:30:13 -07:00
Sean Mackrory
1a25bbe9ec HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota. 2018-12-11 08:33:13 -07:00
Sean Mackrory
c35de95a22 HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota. 2018-12-11 08:29:39 -07:00
Sean Mackrory
3ff8580f22 HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota) 2018-12-10 14:03:08 -07:00
Inigo Goiri
8fc0d04517 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-12-07 18:30:36 -08:00
Da Zhou
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2018-12-04 13:39:10 +00:00
Da Zhou
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.
 2018-11-30 10:12:41 +00:00
Sean Mackrory
7eb0d3a324 HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota. 2018-11-29 09:36:39 -07:00
Ewan Higgs
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.
 2018-11-29 15:12:17 +00:00
Sean Mackrory
085f10e75d HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory
e148c3ff09 HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory
5d96b74f33 HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Steve Loughran
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.
 2018-11-27 20:39:54 +00:00
Da Zhou
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d91aa594283e447963fb2e407d4c6af5)
 2018-11-27 18:13:07 +00:00
Da Zhou
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2018-11-23 14:17:36 +00:00
Weiwei Yang
5ff0cf86a9 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 2018-11-23 13:56:54 +08:00
Weiwei Yang
be0708c6eb HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 2018-11-19 14:00:54 +08:00
Arpit Agarwal
914b0cf15f HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 2018-11-15 13:58:13 -08:00
Akira Ajisaka
66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 2018-11-15 16:30:24 +09:00
Sammi Chen
3fade865ce HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 2018-11-14 12:58:57 +08:00
Da Zhou
a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2018-11-13 21:46:18 +00:00
Da Zhou
9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2018-11-09 11:06:23 +00:00
Junhua Gu
724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9e8f4f25faa352a06d142b75a029f0e)
 2018-11-08 17:21:40 +00:00
Sammi Chen
7574d18538 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 2018-10-26 15:19:56 +08:00
Daniel Templeton
e374584479 HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2018-10-24 14:18:35 -07:00
Robert Kanter
5e294e950b YARN-8919. Some tests fail due to NoClassDefFoundError for OperatorCreationException (tasanuma0829 via rkanter) 2018-10-23 13:37:17 -07:00
Ted Yu
e2cecb681e HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-19 13:21:06 -07:00
Giovanni Matteo Fumarola
cc09b2b0c7 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:24:23 -07:00
Steve Loughran
d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong. 
Contributed by Steve Loughran and Ewan Higgs.
 2018-10-16 20:02:54 +01:00
Steve Loughran
ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console. 
Contributed by Vishwajeet Dusane.
 2018-10-15 22:53:55 +01:00
bibinchundatt
b4a38e7b3e YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt. 2018-10-15 16:10:25 +05:30
Steve Loughran
e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus. 
Contributed by Ted Yu.
 2018-10-12 09:59:19 +01:00
Steve Loughran
ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-11 14:57:38 +01:00
Steve Loughran
bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2018-10-09 20:02:12 +01:00
Jason Lowe
4b5b1ac3d1 MAPREDUCE-7130. Rumen crashes trying to handle MRAppMaster recovery events. Contributed by Peter Bacsko 2018-10-09 13:27:03 -05:00
Steve Loughran
7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard. 
Contributed by Gabor Bota
 2018-10-09 10:46:41 +01:00
Steve Loughran
273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2018-10-05 11:29:03 +01:00
Steve Loughran
7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2018-10-03 12:52:53 +01:00
Aaron Fabbri
4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-03 00:24:29 -07:00
Aaron Fabbri
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-02 21:22:49 -07:00
Steve Loughran
e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2018-10-02 18:57:20 +01:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Steve Loughran
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2018-10-02 11:42:52 +01:00
Steve Loughran
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2018-10-02 11:37:28 +01:00
Steve Loughran
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2018-09-25 20:58:32 +01:00
Steve Loughran
d5da9928c9 HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2018-09-25 19:13:10 +01:00
Sammi Chen
2b635125fb HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 2018-09-25 19:48:30 +08:00
Mingliang Liu
c07715e378 HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran 2018-09-24 12:53:21 -07:00
Sunil G
d060cbea48 HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran. 2018-09-24 21:19:47 +05:30
Sean Mackrory
0def61482b Merge branch 'HADOOP-15407' into trunk 2018-09-22 21:19:12 -06:00
Steve Loughran
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2018-09-21 11:06:24 +01:00
Steve Loughran
a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2018-09-20 17:36:18 +01:00
Sean Mackrory
8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2018-09-19 13:57:39 +01:00
Steve Loughran
df2166a643
HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2018-09-18 12:20:52 +01:00
Steve Loughran
51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2018-09-18 12:09:25 +01:00
Steve Loughran
524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. 
Contributed by Da Zhou
 2018-09-18 11:48:46 +01:00
Steve Loughran
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch. 
Contributed by Steve Loughran.
 2018-09-18 10:56:56 +01:00
Steve Loughran
26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config. 
Contributed by Gabor Bota.
 2018-09-17 22:40:08 +01:00
Thomas Marquardt
b4c23043d3 HADOOP-15757. ABFS: remove dependency on common-codec Base64. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
26211019c8 HADOOP-15753. ABFS: support path "abfs://mycluster/file/path" 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
e5593cbd83 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
13c70e9ba3 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. 
Contributed by Sneha Varma.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
6801b30733 HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation 
for storage account with no namespace feature.

Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
347a52a866 Fixing findbugs and license issues related to: 
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
97f06b3fc7 HADOOP-15703. ABFS - Implement client-side throttling. 
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
4410eacba7 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
81dc4a995c HADOOP-15663. ABFS: Simplify configuration. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
df57c6c3b1 HADOOP-15692. ABFS: extensible support for custom oauth. 
Contributed by Junhua Gu and Rajeev Bansal.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
dd2b22fa31 HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. 
Contributed by Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
6b6f8cc2be HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
9c1e4e8139 HADOOP-15661. ABFS: Add support for ACL. 
Contributed by Junhua Gu and Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
9149b9703e HADOOP-15660. ABFS: Add support for OAuth 
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
d6a4f39bd5 HADOOP-15669. ABFS: Improve HTTPS Performance. 
Contributed by Vishwajeet Dusane.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
cc5cc60c41 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2018-09-17 19:54:01 +00:00
Thomas Marquardt
b54b0c1b67 HADOOP-15659. Code changes for bug fix and new tests. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
ce03a93f78 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. 
Contributed by Steve Loughran and Da Zhou.
 2018-09-17 19:54:01 +00:00
Steve Loughran
a271fd0eca HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Steve Loughran
f044deedbb HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit. 
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
 2018-09-17 19:54:01 +00:00
Steve Loughran
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran) 2018-09-12 21:04:49 -07:00
Aaron Fabbri
d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine) 2018-09-12 16:36:01 -07:00
Mingliang Liu
1f6c4545cf HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran 2018-09-12 10:58:39 -07:00
Sean Mackrory
47b72c87eb HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded. 
Contributed by Gabor Bota.
 2018-09-12 09:12:38 -06:00
bibinchundatt
c44088ac19 YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Naganarasimha
eb08543c7a YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
Sunil G
89b3ebd11e YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Mingliang Liu
87f63b6479 HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran 2018-09-11 17:18:42 -07:00
Steve Loughran
1eef0acfe5
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.
 2018-09-10 14:24:41 +01:00
Gabor Bota
36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota) 2018-09-07 10:25:20 -07:00
Steve Loughran
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.
 2018-08-30 14:49:53 +01:00
Steve Loughran
e8d138ca7c
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.
 2018-08-30 14:36:00 +01:00
Steve Loughran
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length. 
Contributed by Ewan Higgs
 2018-08-30 14:33:16 +01:00
Yufei Gu
8563fd67be YARN-8632. Threads in SLS quit without logging exception. Contributed by Xianghao Lu. 2018-08-24 11:23:17 -07:00
Surendra Singh Lilhore
96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Aaron Fabbri
d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota) 2018-08-17 10:15:39 -07:00
Steve Loughran
0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375. 
Contributed by Steve Loughran.
 2018-08-16 09:58:46 -07:00