big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,122 Commits 2 Branches 2 Tags 264 MiB
f734455e5d
Commit Graph

307 Commits

Author SHA1 Message Date
Steve Loughran
6a859d33aa
HADOOP-16785. followup to abfs close() fix. 
Adds one extra test to the ABFS close logic, to explicitly
verify that the close sequence of FilterOutputStream is
not going to fail.

This is just a due-diligence patch, but it helps ensure
that no regressions creep in in future.

Contributed by Steve Loughran.

Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48
 2020-01-20 16:23:41 +00:00
Clemens Wolff
c36f09deb9
HADOOP-16005. NativeAzureFileSystem does not support setXAttr. 
Contributed by Clemens Wolff.
 2020-01-14 17:28:37 -08:00
Steve Loughran
17aa8f6764
HADOOP-16785. Improve wasb and abfs resilience on double close() calls. 
This hardens the wasb and abfs output streams' resilience to being invoked
in/after close().

wasb:
  Explicity raise IOEs on operations invoked after close,
  rather than implicitly raise NPEs.
  This ensures that invocations which catch and swallow IOEs will perform as
  expected.

abfs:
  When rethrowing an IOException in the close() call, explicitly wrap it
  with a new instance of the same subclass.
  This is needed to handle failures in try-with-resources clauses, where
  any exception in closed() is added as a suppressed exception to the one
  thrown in the try {} clause
  *and you cannot attach the same exception to itself*

Contributed by Steve Loughran.

Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b
 2020-01-08 11:46:54 +00:00
Sneha Vijayarajan
d1f5976c00
HADOOP-16699. Add verbose TRACE logging to ABFS. 
Contributed by Sneha Vijayarajan,

Change-Id: Ic616a10406e6e9f11616c9cc05d8630ebbedaf65
 2020-01-07 18:05:47 +00:00
Akira Ajisaka
f777cd398f
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor. 2019-12-20 13:10:26 +09:00
Sneha Vijayarajan
82ad9b549f
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable. 
Contributed by Sneha Vijayarajan.
 2019-11-27 15:10:21 -08:00
bilaharith
9e69628f55 HADOOP-16455. ABFS: Implement FileSystem.access() method. 
Contributed by Bilahari T H.
 2019-11-27 15:56:38 +00:00
Sneha Vijayarajan
de38045021 HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account. (#1701) 2019-11-21 11:24:12 +09:00
Jeetesh Mangwani
b033c681e4
HADOOP-16612. Track Azure Blob File System client-perceived latency 
Contributed by Jeetesh Mangwani.

This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.
The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.
 2019-11-19 09:00:24 -08:00
Andras Bokor
96c4520f89
HADOOP-16710. Testing_azure.md documentation is misleading. 
Contributed by Andras Bokor.

Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
 2019-11-17 17:04:29 +00:00
Da Zhou
9a8edb0aed
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 2019-10-16 13:14:15 -07:00
bilaharith
1a77a15fe4
HADOOP-16587. Make ABFS AAD endpoints configurable. 
Contributed by Bilahari T H.

This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.

Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
 2019-10-07 13:07:46 +01:00
Sneha Vijayarajan
770adc5d4a
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists 2019-10-01 17:38:11 -07:00
Sneha Vijayarajan
c0edc848a8
HADOOP-16548 : Disable Flush() over config 2019-09-28 20:39:42 -07:00
Steve Loughran
e346e3638c HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.

Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
 2019-09-25 12:16:41 +01:00
Gabor Bota
aa664d7259
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE (#1302). Contributed by Gabor Bota. 
Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023
 2019-09-23 13:29:01 +02:00
Sahil Takiar
55ce454ce4
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8. 
Contributed by Sahil Takiar.

This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common
as the DelegatingSSLSocketFactory and binds the S3A connector to it so that
it can avoid using those HTTPS algorithms which are underperformant on Java 8.

Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd
 2019-09-17 11:32:03 +01:00
Da Zhou
43a91f820a
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-09 12:38:13 +01:00
bilaharith
5840df86d7
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-08 19:08:48 +01:00
Arun Singh
0b45293abb
HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-07-19 20:21:28 -07:00
Robert Levas
450c070a8f
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations. 
Contributed by Robert Levas.

This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.

Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
 2019-06-19 17:43:14 +01:00
Da Zhou
1da09bd9d5
HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-06-16 19:20:46 +01:00
Steve Loughran
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" 
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
 2019-06-05 13:54:55 +01:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Sahil Takiar
b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)
 2019-05-16 08:57:54 -06:00
DadanielZ
ff27e8eabd HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata. 
Author: Da Zhou
 2019-05-10 13:57:30 -07:00
Aaron Fabbri
5f1a34649f Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
 2019-05-10 13:55:56 -07:00
DadanielZ
7c2d7c05a9 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-05-10 12:46:03 -07:00
Da Zhou
3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-05-08 17:20:46 +01:00
Da Zhou
1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-04-29 13:27:28 +01:00
Akira Ajisaka
4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19 2019-04-15 10:46:10 +09:00
Akira Ajisaka
8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 2019-04-03 19:35:02 +09:00
Vishwajeet Dusane
1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.
 2019-03-18 13:18:08 +00:00
Denes Gerencser
ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console. 
Follow-on to HADOOP-15851.

Author:    Denes Gerencser <dgerencser@cloudera.com>
 2019-03-09 18:45:17 +00:00
Shweta Yakkali
bb8ad096e7 HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
 2019-03-09 09:55:56 -08:00
Da Zhou
e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-03-08 13:53:08 +00:00
Da Zhou
05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens. 
Contributed by Da Zhou.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:02:34 +00:00
Steve Loughran
65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store. 
Contributed by Steve Loughran.
 2019-02-28 14:22:32 +00:00
Da Zhou
3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.
 2019-02-26 15:37:24 +00:00
Steve Loughran
a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.
 2019-02-21 14:08:46 +00:00
Masatake Iwasaki
aa3ad36605 HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki. 2019-02-20 22:00:57 +09:00
Da Zhou
1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-07 21:58:21 +00:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Da Zhou
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code. 
Contributed by Da Zhou.
 2019-02-05 14:39:45 +00:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Shweta Yakkali
02eb91856e HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali. 
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
 2019-01-29 16:50:14 -08:00
Da Zhou
00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.
 2019-01-22 16:40:42 +00:00
Da Zhou
30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.
 2019-01-14 19:32:56 +00:00
Da Zhou
dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.

(cherry picked from commit 8b5fbe7a125f9d08cbb9f5e5ae28dc984e0d73d8)
 2019-01-11 11:13:41 +00:00
Da Zhou
852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement. 
Contributed by Da Zhou.
 2019-01-10 12:07:09 +00:00
Da Zhou
7211269142
HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.
 2019-01-10 12:02:58 +00:00
Da Zhou
e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.
 2019-01-10 11:58:39 +00:00
Sean Mackrory
13f0ee21f2 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. 2019-01-02 09:33:09 -07:00
Da Zhou
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.
 2018-12-17 11:15:20 +00:00
Da Zhou
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb6889e51d4d4c914eb6159e40845e9f)
 2018-12-17 11:10:22 +00:00
Da Zhou
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2018-12-17 11:04:40 +00:00
Da Zhou
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2018-12-04 13:39:10 +00:00
Da Zhou
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.
 2018-11-30 10:12:41 +00:00
Da Zhou
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d91aa594283e447963fb2e407d4c6af5)
 2018-11-27 18:13:07 +00:00
Da Zhou
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2018-11-23 14:17:36 +00:00
Da Zhou
a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2018-11-13 21:46:18 +00:00
Da Zhou
9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2018-11-09 11:06:23 +00:00
Junhua Gu
724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9e8f4f25faa352a06d142b75a029f0e)
 2018-11-08 17:21:40 +00:00
Daniel Templeton
e374584479 HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2018-10-24 14:18:35 -07:00
Steve Loughran
ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console. 
Contributed by Vishwajeet Dusane.
 2018-10-15 22:53:55 +01:00
Steve Loughran
bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2018-10-09 20:02:12 +01:00
Steve Loughran
273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2018-10-05 11:29:03 +01:00
Steve Loughran
7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2018-10-03 12:52:53 +01:00
Steve Loughran
e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2018-10-02 18:57:20 +01:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Steve Loughran
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2018-10-02 11:42:52 +01:00
Steve Loughran
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2018-10-02 11:37:28 +01:00
Steve Loughran
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2018-09-25 20:58:32 +01:00
Steve Loughran
d5da9928c9 HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2018-09-25 19:13:10 +01:00
Steve Loughran
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2018-09-21 11:06:24 +01:00
Steve Loughran
a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2018-09-20 17:36:18 +01:00
Sean Mackrory
8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2018-09-19 13:57:39 +01:00
Steve Loughran
df2166a643
HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2018-09-18 12:20:52 +01:00
Steve Loughran
51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2018-09-18 12:09:25 +01:00
Steve Loughran
524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. 
Contributed by Da Zhou
 2018-09-18 11:48:46 +01:00
Steve Loughran
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch. 
Contributed by Steve Loughran.
 2018-09-18 10:56:56 +01:00
Thomas Marquardt
b4c23043d3 HADOOP-15757. ABFS: remove dependency on common-codec Base64. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
26211019c8 HADOOP-15753. ABFS: support path "abfs://mycluster/file/path" 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
e5593cbd83 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
13c70e9ba3 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. 
Contributed by Sneha Varma.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
6801b30733 HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation 
for storage account with no namespace feature.

Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
347a52a866 Fixing findbugs and license issues related to: 
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
97f06b3fc7 HADOOP-15703. ABFS - Implement client-side throttling. 
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
4410eacba7 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
81dc4a995c HADOOP-15663. ABFS: Simplify configuration. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
df57c6c3b1 HADOOP-15692. ABFS: extensible support for custom oauth. 
Contributed by Junhua Gu and Rajeev Bansal.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
dd2b22fa31 HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. 
Contributed by Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
6b6f8cc2be HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
9c1e4e8139 HADOOP-15661. ABFS: Add support for ACL. 
Contributed by Junhua Gu and Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
9149b9703e HADOOP-15660. ABFS: Add support for OAuth 
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
d6a4f39bd5 HADOOP-15669. ABFS: Improve HTTPS Performance. 
Contributed by Vishwajeet Dusane.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt
cc5cc60c41 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2018-09-17 19:54:01 +00:00