big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
23,861 Commits 2 Branches 2 Tags 264 MiB
439c51425e
Commit Graph

1361 Commits

Author SHA1 Message Date
Akira Ajisaka
567091aa9b
HADOOP-15958. Revisiting LICENSE and NOTICE files. 
This closes #1307

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-27 13:47:12 +09:00
Ewan Higgs
23e532d739 Revert "HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran" 
This reverts commit 69ddb36876.
 2019-08-26 12:37:26 +02:00
Akira Ajisaka
84b1982060
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 2019-08-23 14:39:31 +09:00
Erik Krogen
63c295e298 HDFS-14755. [Dynamometer] Enhance compatibility of Dynamometer with branch-2 builds. Contributed by Takanobu Asanuma. 2019-08-22 09:57:12 -07:00
Steve Loughran
61b2df2331
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper. 
Contributed by Steve Loughran.

Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.

Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8
 2019-08-22 17:27:56 +01:00
Ewan Higgs
69ddb36876 HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran 2019-08-22 13:56:47 +02:00
Takanobu Asanuma
ee7c261e1e
HDFS-14763. Fix package name of audit log class in Dynamometer document (#1335) 2019-08-22 18:37:16 +09:00
Abhishek Modi
3ad1fcfc8b YARN-9752. Add support for allocation id in SLS. Contributed by Abhishek Modi 2019-08-21 20:39:51 +05:30
bibinchundatt
10ec31d20e YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 2019-08-21 13:48:21 +05:30
KAI XIE
c765584eb2 HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode
 2019-08-18 18:46:31 -07:00
Steve Loughran
0e4b757955 HADOOP-16500 S3ADelegationTokens to only log at debug on startup (#1269). Contributed by Steve Loughran. 
Change-Id: Ifafc15f32791911976d7ebc36fb6e8853f59ed41
 2019-08-14 10:50:26 +02:00
Erik Krogen
274966e675 HDFS-14717. [Dynamometer] Remove explicit search for JUnit dependency JAR from Dynamometer Client as it is packaged in the primary JAR. Contributed by Kevin Su. 2019-08-13 08:52:59 -07:00
Steve Loughran
189dc10884 HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region. (#1209). Contributed by Steve Loughran. 2019-08-09 17:33:08 +02:00
Steve Loughran
e25a5c2eab HADOOP-16499. S3A retry policy to be exponential (#1246). Contributed by Steve Loughran. 2019-08-09 15:52:37 +02:00
Da Zhou
43a91f820a
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-09 12:38:13 +01:00
Abhishek Modi
a92b7a5491 YARN-9694. UI always show default-rack for all the nodes while running SLS. 2019-08-09 11:41:16 +05:30
bilaharith
5840df86d7
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-08 19:08:48 +01:00
Steve Loughran
b01efe5cf6
HADOOP-16472. findbugs warning on LocalMetadataStore.ttlTimeProvider sync 
Contributed by Steve Loughran.

Moved the setter and addAncestors to synchronized

Change-Id: Ib362c66d1b8c9124eca7db9a44274ac08d0b3be6
 2019-08-02 22:30:48 +01:00
Felipe Lopes
bca86bd289
HADOOP-16469. Update committers.md 
Contributed by Felipe Lopes.

Change-Id: I5c05b878bde073aeb45bf22340183893f85269e1
 2019-07-30 12:47:55 +01:00
Gabor Bota
7b219778e0
HADOOP-16433. S3Guard: Filter expired entries and tombstones when listing with MetadataStore.listChildren(). 
Contributed by Gabor Bota.

This pulls the tracking of the lastUpdated timestamp of metadata entries up from the DDB metastore into all s3guard stores, and then uses this to filter out expired tombstones from listings.

Change-Id: I80f121236b49c75a024116f65a3ef29d3580b462
 2019-07-24 18:11:43 +01:00
sunlisheng
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java. 
This closes #1091

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-07-24 10:04:39 +09:00
Steve Loughran
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories. 
Contributed by Steve Loughran and Gabor Bota.

This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.

It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".

Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
 2019-07-23 14:52:03 +01:00
Ayush Saxena
e60f5e2572 HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun. 2019-07-20 13:11:14 +05:30
Arun Singh
0b45293abb
HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-07-19 20:21:28 -07:00
Sean Mackrory
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration (#1125) 2019-07-19 09:49:59 -06:00
lqjaclee
cd967c75a7
HADOOP-15847. S3Guard testConcurrentTableCreations to set R/W capacity == 0 
Contributed by lqjaclee

Change-Id: I4a4d5b29f2677c188799479e4db38f07fa0591d1
 2019-07-19 14:46:55 +01:00
Josh Rosen
d545f9c290 HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise 
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise 

Contributed by: Josh Rosen
 2019-07-18 23:19:38 +01:00
Gabor Bota
c58e11bf52
HADOOP-16383. Pass ITtlTimeProvider instance in initialize method in MetadataStore interface. Contributed by Gabor Bota. (#1009) 2019-07-17 16:24:39 +02:00
Steve Loughran
19a001826f
Revert "HDFS-9913. DistCp to add -useTrash to move deleted files to Trash." 
Reverting due to test failures if ~/.Trash not present during test setup.

This reverts commit ee3115f488.

Change-Id: Icbeeb261570b9131ff99d765ac0945c335b26658
 2019-07-17 13:13:24 +01:00
Shen Yinjie
ee3115f488
HDFS-9913. DistCp to add -useTrash to move deleted files to Trash. 
Contributed by Shen Yinjie.

Change-Id: I03ac7d22ab1054f8e5de4aa7552909c734438f4a
 2019-07-17 11:50:46 +01:00
Sean Mackrory
5672efa5c7
HADOOP-15729. [s3a] Allow core threads to time out. (#1075) 2019-07-16 18:14:23 -06:00
Steve Loughran
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3. 
Contributed by Steve Loughran

Contains

- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently

This patch doesn't fix the underlying problem but it

* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones

The outcome is that tests should pass consistently and if problems occur we have better diagnostics.

Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
 2019-07-12 13:02:25 +01:00
Steve Loughran
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists. 
Contributed by Steve Loughran.

This patch

* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.

Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
 2019-07-11 18:15:34 +01:00
Yiqun Lin
5043840b1d HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen. 2019-07-11 23:47:27 +08:00
Erik Krogen
32925d04d9 HDFS-14640. [Dynamometer] Fix TestDynamometerInfra failure. Contributed by Erik Krogen. 2019-07-11 08:34:39 -07:00
Erik Krogen
fc0656dd30 HADOOP-16418. [Dynamometer] Fix checkstyle and findbugs warnings. Contributed by Erik Krogen. 2019-07-11 08:29:57 -07:00
Steve Loughran
c7b5f858a0
HADOOP-16393. S3Guard init command uses global settings, not those of target bucket. 
Contributed by Steve Loughran.

Change-Id: I226a91ab8d7758340f8d221aa80a7abf9a0d3e8f
 2019-07-10 20:57:02 +01:00
Erik Krogen
90b10a0d54 HDFS-14622. [Dynamometer] Update XML FsImage parsing logic to ignore non-INodeSection entries to fix an issue caused by the presence of Centralized Cache Management functionality. Contributed by Erik Krogen. 2019-07-10 09:59:11 -07:00
Masatake Iwasaki
738c09349e HADOOP-16411. Fix javadoc warnings in hadoop-dynamometer. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-07-09 09:44:49 +09:00
Sean Mackrory
de6b7bc67a HADOOP-16409. Allow authoritative mode on non-qualified paths. Contributed by Sean Mackrory 2019-07-08 19:27:07 +02:00
Sean Mackrory
34747c373f
HADOOP-16396. Allow authoritative mode on a subdirectory. (#1043) 2019-07-03 12:04:47 -06:00
Erik Krogen
ab0b180ddb HDFS-12345 Add Dynamometer to hadoop-tools, a tool for scale testing the HDFS NameNode with real metadata and workloads. Contributed by Erik Krogen. 2019-06-25 08:07:39 -07:00
kkori
366f3deec5 HADOOP-16390. escape javadoc in S3AUtils public methods 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-06-25 17:47:37 +09:00
Takanobu Asanuma
98d2065643 HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:17:24 -07:00
Steve Loughran
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename. 
Contributed by Steve Loughran.

Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
 2019-06-20 09:56:40 +01:00
Sahil Takiar
28291a9e8a
HADOOP-16379: S3AInputStream.unbuffer should merge input stream stats into fs-wide stats 
Contributed by Sahil Takiar

Change-Id: I2bcfaaea00d12c633757069402dcd0b91a5f5c05
 2019-06-20 09:42:27 +01:00
Robert Levas
450c070a8f
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations. 
Contributed by Robert Levas.

This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.

Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
 2019-06-19 17:43:14 +01:00
Da Zhou
1da09bd9d5
HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-06-16 19:20:46 +01:00
Gabor Bota
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones). 
Contributed by Gabor Bota.

Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
 2019-06-16 17:05:01 +01:00
Wanqiang Ji
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to mapreduce.job.userlog.retain.hours 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-11 18:40:35 +09:00
Steve Loughran
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables. 
Contributed by Steve Loughran

Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
 2019-06-07 18:26:10 +01:00
Akira Ajisaka
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Steve Loughran
ec26c431f9
HADOOP-16117. Update AWS SDK to 1.11.563. 
Contributed by Steve Loughran.

Change-Id: I7c46ed2a6378e1370f567acf4cdcfeb93e43fa13
 2019-06-06 10:08:18 +01:00
Steve Loughran
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL" 
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
 2019-06-05 13:54:55 +01:00
Steve Loughran
7724d8031b Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors" 
This reverts commit 5906268f0d.
 2019-06-05 13:54:42 +01:00
Akira Ajisaka
afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Steve Loughran
0c73dba3a6
HADOOP-16332. Remove S3A dependency on http core. 
Contributed by Steve Loughran.

Change-Id: I53209c993a405fefdb5e1b692d5a56d027d3b845
 2019-05-28 22:50:37 +01:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Sahil Takiar
5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ben Roling
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite. 
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
 2019-05-19 22:29:54 +01:00
Sahil Takiar
b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL 
(cherry picked from commit aebf229c175dfa19fff3b31e9e67596f6c6124fa)
 2019-05-16 08:57:54 -06:00
Andrew Olson
c15b3bca86
HADOOP-16294: Enable access to input options by DistCp subclasses. 
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
 2019-05-16 16:11:12 +02:00
Weiwei Yang
2d8282bb82 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 2019-05-14 14:05:39 -07:00
DadanielZ
ff27e8eabd HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata. 
Author: Da Zhou
 2019-05-10 13:57:30 -07:00
Aaron Fabbri
5f1a34649f Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
 2019-05-10 13:55:56 -07:00
DadanielZ
7c2d7c05a9 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-05-10 12:46:03 -07:00
Rajat Khandelwal
591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die. 
Contributed by Rajat Khandelwal
 2019-05-09 21:13:29 +01:00
Da Zhou
3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-05-08 17:20:46 +01:00
Giovanni Matteo Fumarola
7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Ben Roling
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure. 2019-04-30 11:53:26 +01:00
Da Zhou
1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-04-29 13:27:28 +01:00
Ben Roling
e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests. 
Contributed by Ben Roling.
 2019-04-24 14:55:58 +01:00
Takanobu Asanuma
a5ceed26f3 HADOOP-16253. Update AssertJ to 3.12.2. 2019-04-16 12:53:52 +09:00
Akira Ajisaka
4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19 2019-04-15 10:46:10 +09:00
Sahil Takiar
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer. 
Author:    Sahil Takiar <stakiar@cloudera.com>
 2019-04-12 18:12:02 -07:00
Masatake Iwasaki
bbdbc7a9a1 HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 2019-04-12 11:52:18 +09:00
Steve Loughran
cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables. 
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.

Contributed by Steve Loughran.
 2019-04-11 17:12:12 -07:00
Giovanni Matteo Fumarola
358e928622 YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Addendum. Contributed by Inigo Goiri. 2019-04-09 14:18:05 -07:00
Steve Loughran
215ffc792e HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException 
Contributed by Steve Loughran.

Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
 2019-04-04 21:14:18 +01:00
Steve Loughran
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:23:40 +01:00
Akira Ajisaka
8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 2019-04-03 19:35:02 +09:00
Steve Loughran
df578c07ec HADOOP-16195 MarshalledCredentials toString 
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
 2019-03-28 17:01:57 +00:00
Gabor Bota
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations. 
Author:    Gabor Bota
 2019-03-28 15:59:25 +00:00
Gabor Bota
cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren. 
Author:    Gabor Bota
 2019-03-28 15:49:56 +00:00
Siyao Meng
ce4bafdf44
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail. 
Contributed by Siyao Meng
 2019-03-26 18:42:54 +00:00
Lokesh Jain
ae2eb2dd42 HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists (#636) 2019-03-22 11:42:00 +00:00
Andrew Olson
faba3591d3
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized. 
Author:    Andrew Olson
 2019-03-22 10:35:30 +00:00
Steve Loughran
9f1c017f44
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-21 11:15:37 +00:00
Ajay Kumar
399563fec6 HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia. 2019-03-20 11:20:45 -07:00
Weiwei Yang
568d3ab8b6 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 2019-03-19 09:55:32 +08:00
Adam Antal
c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants. 
Contributed by Adam Antal.
 2019-03-18 19:13:13 +00:00
Vishwajeet Dusane
1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.
 2019-03-18 13:18:08 +00:00
Ben Roling
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-13 20:37:11 +00:00
Denes Gerencser
ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console. 
Follow-on to HADOOP-15851.

Author:    Denes Gerencser <dgerencser@cloudera.com>
 2019-03-09 18:45:17 +00:00
Shweta Yakkali
bb8ad096e7 HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
 2019-03-09 09:55:56 -08:00
Steve Loughran
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-09 16:00:34 +00:00
Da Zhou
e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-03-08 13:53:08 +00:00
Da Zhou
05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens. 
Contributed by Da Zhou.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:02:34 +00:00
Steve Loughran
65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store. 
Contributed by Steve Loughran.
 2019-02-28 14:22:32 +00:00
Abhishek Modi
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util. 
Contributed by Abhishek Modi
 2019-02-26 17:10:41 +00:00
Da Zhou
3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.
 2019-02-26 15:37:24 +00:00
Steve Loughran
a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.
 2019-02-21 14:08:46 +00:00
Masatake Iwasaki
aa3ad36605 HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki. 2019-02-20 22:00:57 +09:00
bibinchundatt
779dae4de7 YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 2019-02-20 00:36:57 +05:30
Adam Antal
1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found. 
Contributed by Adam Antal.

(Revised patch applied after stevel committed the wrong one; that has been reverted)
 2019-02-19 11:33:02 +00:00
Steve Loughran
920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found." 
This reverts commit c4a00d1ad3.
 2019-02-18 14:57:22 +00:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
bibinchundatt
134ae8fc80 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 2019-02-14 22:56:52 +05:30
Masatake Iwasaki
6c999fe4b0 HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki. 2019-02-12 06:07:47 +09:00
Da Zhou
1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-07 21:58:21 +00:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth." 
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
 2019-02-07 21:57:22 +00:00
Ranith Sardar
546c5d70ef
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 2019-02-07 21:48:07 +00:00
Andrew Olson
de804e53b9
HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.
 2019-02-07 10:07:22 +00:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-02-05 19:23:15 +00:00
Da Zhou
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code. 
Contributed by Da Zhou.
 2019-02-05 14:39:45 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile(); 
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
 2019-02-05 11:51:02 +00:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Shweta Yakkali
02eb91856e HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali. 
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
 2019-01-29 16:50:14 -08:00
Akira Ajisaka
3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document. 2019-01-25 10:27:59 +09:00
Da Zhou
00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.
 2019-01-22 16:40:42 +00:00
Da Zhou
30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.
 2019-01-14 19:32:56 +00:00
Steve Loughran
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran and Daryn Sharp.
 2019-01-14 17:59:27 +00:00
Adam Antal
c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found. 
Contributed by Adam Antal.
 2019-01-14 17:27:00 +00:00
Gabor Bota
04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set. 
Contributed by Gabor Bota.
 2019-01-14 13:12:05 +00:00
Giovanni Matteo Fumarola
fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
Da Zhou
dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.

(cherry picked from commit 8b5fbe7a125f9d08cbb9f5e5ae28dc984e0d73d8)
 2019-01-11 11:13:41 +00:00
Da Zhou
852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement. 
Contributed by Da Zhou.
 2019-01-10 12:07:09 +00:00
Da Zhou
7211269142
HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.
 2019-01-10 12:02:58 +00:00
Da Zhou
e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.
 2019-01-10 11:58:39 +00:00
Sean Mackrory
3420e26ae5 HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota. 2019-01-09 10:57:58 -07:00
Kai Xie
188bebbe7e HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-01-08 11:57:57 +00:00
Akira Ajisaka
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens." 
This reverts commit d7152332b3.
 2019-01-08 14:51:30 +09:00
Steve Loughran
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran.
 2019-01-07 13:18:03 +00:00
Weiwei Yang
f87b3b11c4 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 2019-01-07 15:56:49 +08:00
Sunil G
1f425271a7
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae4.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb.
 2019-01-05 17:35:29 +09:00
Weiwei Yang
040a202b20 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 2019-01-03 21:37:49 +08:00
Sean Mackrory
13f0ee21f2 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. 2019-01-02 09:33:09 -07:00
Sean Mackrory
d8f670ff28 HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal. 2018-12-27 08:19:25 -07:00
Eric Yang
f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Steve Loughran
a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies. 
Contributed by Steve Loughran.
 2018-12-20 18:09:01 +00:00
Vishwajeet Dusane
784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress. 
Contributed by Vishwajeet Dusane.
 2018-12-20 18:00:05 +00:00
Da Zhou
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.
 2018-12-17 11:15:20 +00:00
Da Zhou
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb6889e51d4d4c914eb6159e40845e9f)
 2018-12-17 11:10:22 +00:00
Da Zhou
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2018-12-17 11:04:40 +00:00
Sean Mackrory
82b798581d HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota. 2018-12-12 09:30:13 -07:00
Sean Mackrory
1a25bbe9ec HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota. 2018-12-11 08:33:13 -07:00
Sean Mackrory
c35de95a22 HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota. 2018-12-11 08:29:39 -07:00
Sean Mackrory
3ff8580f22 HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota) 2018-12-10 14:03:08 -07:00
Inigo Goiri
8fc0d04517 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-12-07 18:30:36 -08:00
Da Zhou
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2018-12-04 13:39:10 +00:00
Da Zhou
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.
 2018-11-30 10:12:41 +00:00
Sean Mackrory
7eb0d3a324 HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota. 2018-11-29 09:36:39 -07:00
Ewan Higgs
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.
 2018-11-29 15:12:17 +00:00
Sean Mackrory
085f10e75d HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory
e148c3ff09 HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory
5d96b74f33 HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Steve Loughran
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.
 2018-11-27 20:39:54 +00:00
Da Zhou
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d91aa594283e447963fb2e407d4c6af5)
 2018-11-27 18:13:07 +00:00
Da Zhou
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2018-11-23 14:17:36 +00:00
Weiwei Yang
5ff0cf86a9 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 2018-11-23 13:56:54 +08:00
Weiwei Yang
be0708c6eb HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 2018-11-19 14:00:54 +08:00
Arpit Agarwal
914b0cf15f HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 2018-11-15 13:58:13 -08:00
Akira Ajisaka
66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 2018-11-15 16:30:24 +09:00
Sammi Chen
3fade865ce HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 2018-11-14 12:58:57 +08:00
Da Zhou
a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2018-11-13 21:46:18 +00:00
Da Zhou
9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2018-11-09 11:06:23 +00:00
Junhua Gu
724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9e8f4f25faa352a06d142b75a029f0e)
 2018-11-08 17:21:40 +00:00
Sammi Chen
7574d18538 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 2018-10-26 15:19:56 +08:00
Daniel Templeton
e374584479 HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2018-10-24 14:18:35 -07:00
Robert Kanter
5e294e950b YARN-8919. Some tests fail due to NoClassDefFoundError for OperatorCreationException (tasanuma0829 via rkanter) 2018-10-23 13:37:17 -07:00
Ted Yu
e2cecb681e HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-19 13:21:06 -07:00
Giovanni Matteo Fumarola
cc09b2b0c7 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:24:23 -07:00
Steve Loughran
d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong. 
Contributed by Steve Loughran and Ewan Higgs.
 2018-10-16 20:02:54 +01:00
Steve Loughran
ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console. 
Contributed by Vishwajeet Dusane.
 2018-10-15 22:53:55 +01:00
bibinchundatt
b4a38e7b3e YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt. 2018-10-15 16:10:25 +05:30
Steve Loughran
e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus. 
Contributed by Ted Yu.
 2018-10-12 09:59:19 +01:00
Steve Loughran
ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-11 14:57:38 +01:00
Steve Loughran
bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2018-10-09 20:02:12 +01:00
Jason Lowe
4b5b1ac3d1 MAPREDUCE-7130. Rumen crashes trying to handle MRAppMaster recovery events. Contributed by Peter Bacsko 2018-10-09 13:27:03 -05:00
Steve Loughran
7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard. 
Contributed by Gabor Bota
 2018-10-09 10:46:41 +01:00
Steve Loughran
273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2018-10-05 11:29:03 +01:00
Steve Loughran
7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2018-10-03 12:52:53 +01:00
Aaron Fabbri
4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-03 00:24:29 -07:00
Aaron Fabbri
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-02 21:22:49 -07:00
Steve Loughran
e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2018-10-02 18:57:20 +01:00
Sunil G
58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Steve Loughran
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2018-10-02 11:42:52 +01:00
Steve Loughran
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2018-10-02 11:37:28 +01:00
Steve Loughran
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2018-09-25 20:58:32 +01:00
Steve Loughran
d5da9928c9 HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2018-09-25 19:13:10 +01:00
Sammi Chen
2b635125fb HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 2018-09-25 19:48:30 +08:00
Mingliang Liu
c07715e378 HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran 2018-09-24 12:53:21 -07:00
Sunil G
d060cbea48 HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran. 2018-09-24 21:19:47 +05:30
Sean Mackrory
0def61482b Merge branch 'HADOOP-15407' into trunk 2018-09-22 21:19:12 -06:00
Steve Loughran
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2018-09-21 11:06:24 +01:00
Steve Loughran
a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2018-09-20 17:36:18 +01:00
Sean Mackrory
8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2018-09-19 13:57:39 +01:00
Steve Loughran
df2166a643
HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2018-09-18 12:20:52 +01:00
Steve Loughran
51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2018-09-18 12:09:25 +01:00