Commit Graph

1508 Commits

Author SHA1 Message Date
Steve Loughran
2dd1e04010
HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937)
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
2021-04-20 15:32:01 +01:00
billierinaldi
c1fde4fe94
HADOOP-16948. Support infinite lease dirs. (#1925)
* HADOOP-16948. Support single writer dirs.

* HADOOP-16948. Fix findbugs and checkstyle problems.

* HADOOP-16948. Fix remaining checkstyle problems.

* HADOOP-16948. Add DurationInfo, retry policy for acquiring lease, and javadocs

* HADOOP-16948. Convert ABFS client to use an executor for lease ops

* HADOOP-16948. Fix ABFS lease test for non-HNS

* HADOOP-16948. Fix checkstyle and javadoc

* HADOOP-16948. Address review comments

* HADOOP-16948. Use daemon threads for ABFS lease ops

* HADOOP-16948. Make lease duration configurable

* HADOOP-16948. Add error messages to test assertions

* HADOOP-16948. Remove extra isSingleWriterKey call

* HADOOP-16948. Use only infinite lease duration due to cost of renewal ops

* HADOOP-16948. Remove acquire/renew/release lease methods

* HADOOP-16948. Rename single writer dirs to infinite lease dirs

* HADOOP-16948. Fix checkstyle

* HADOOP-16948. Wait for acquire lease future

* HADOOP-16948. Add unit test for acquire lease failure
2021-04-12 19:47:59 -04:00
sumangala-patki
6f640abbaf
HADOOP-17576. ABFS: Disable throttling update for auth failures (#2761)
Contributed by Sumangala Patki
2021-04-09 09:31:23 +05:30
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 13:39:10 +09:00
Steve Loughran
85d3bba555
HADOOP-17476. ITestAssumeRole.testAssumeRoleBadInnerAuth failure. (#2777)
Contributed by Steve Loughran.
2021-03-24 16:47:55 +00:00
Steve Loughran
04880f076d
HADOOP-13551. AWS metrics wire-up (#2778)
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.

This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.

Explicitly setting the endpoint will save the cost of these probe
HTTP requests.

Contributed by Steve Loughran.
2021-03-24 13:32:54 +00:00
Ayush Saxena
03cfc85279
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732). Contributed by Ayush Saxena.
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-24 02:36:26 +05:30
Jack Jiang
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Ayush Saxena
4781761dc2
HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-19 14:19:49 +05:30
Chao Sun
9b2f812996 HADOOP-17532. Yarn Job execution get failed when LZ4 Compression Codec is used. Contributed Bhavik Patel. 2021-03-14 21:15:08 -07:00
sumangala-patki
fe633d4739
HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter (#2729)
Contributed by Sumangala Patki.
2021-03-14 13:35:02 +05:30
Steve Loughran
bcd9c67082
HADOOP-16721. Improve S3A rename resilience (#2742)
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
2021-03-11 12:47:39 +00:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753)
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
Pierrick Hymbert
ebfba0b6fa
[HADOOP-17567] typo in MagicCommitTracker (#2749)
Contributed by Pierrick Hymbert
2021-03-10 15:39:55 +00:00
Chao Sun
176bd88890
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. (#2522)
Contributed by Chao Sun.

(Cherry-picked via PR #2575)
2021-03-09 20:01:29 +00:00
Peter Bacsko
c3aa413ee3 YARN-10679. Better logging of uncaught exceptions throughout SLS. Contributed by Szilard Nemeth. 2021-03-09 14:02:12 +01:00
Peter Bacsko
099f58f8f4 YARN-10681. Fix assertion failure message in BaseSLSRunnerTest. Contributed by Szilard Nemeth. 2021-03-09 13:22:48 +01:00
Peter Bacsko
7f522c92fa YARN-10677. Logger of SLSFairScheduler is provided with the wrong class. Contributed by Szilard Nemeth. 2021-03-09 12:53:32 +01:00
Peter Bacsko
ea90cd3556 YARN-10678. Try blocks without catch blocks in SLS scheduler classes can swallow other exceptions. Contributed by Szilard Nemeth. 2021-03-09 12:03:53 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
sumangala-patki
7f64030314
HADOOP-17537. ABFS: Correct assertion reversed in HADOOP-13327
Contributed Sumangala Patki.
2021-02-22 11:45:58 +00:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)"
This reverts commit 4cf3531583.
2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
Anoop Sam John
1bb4101b59
HADOOP-17038 Support disabling buffered reads in ABFS positional reads. (#2646)
- Contributed by @anoopsjohn
2021-02-16 22:27:52 +05:30
Steve Loughran
78905d7e3f
HADOOP-16906. Abortable (#2684)
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.
2021-02-11 17:37:20 +00:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. (#2587)
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
bilaharith
5f34271bb1
HADOOP-17475. ABFS : add high performance listStatusIterator (#2548)
The ABFS connector now implements listStatusIterator() with
asynchronous prefetching of the next page(s) of results.
For listing large directories this can provide tangible speedups.

If for any reason this needs to be disabled, set
fs.azure.enable.abfslistiterator to false.

Contributed by Bilahari T H.
2021-02-04 13:36:19 +00:00
Steve Loughran
26b9d480e8
HADOOP-17337. S3A NetworkBinding has a runtime dependency on shaded httpclient. (#2599)
Contributed by Steve Loughran.
2021-02-03 14:29:56 +00:00
Steve Loughran
f37bf65199
HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException. (#2648)
When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.
2021-02-02 18:13:41 +00:00
Steve Loughran
0bb52a42e5
HADOOP-17483. Magic committer is enabled by default. (#2656)
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
2021-01-27 19:04:22 +00:00
Steve Loughran
28cc912a5c
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649)
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.
2021-01-27 16:39:29 +00:00
Steve Loughran
80c7404b51
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530)
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.
2021-01-26 19:30:51 +00:00
Ayush Saxena
e40f99f6d5 HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun. 2021-01-26 14:25:27 +05:30
Steve Loughran
06a5d3437f
HADOOP-17480. Document that AWS S3 is consistent and that S3Guard is not needed (#2636)
Contributed by Steve Loughran.
2021-01-25 13:21:34 +00:00
Maksim Bober
e2f8503ebd
HADOOP-17484. Typo in hadop-aws index.md (#2634)
Contributed by Maksim Bober.
2021-01-21 17:30:58 +00:00
Steve Loughran
68bc721841
HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole. (#2600)
Contributed by Steve Loughran.
2021-01-19 17:19:27 +00:00
Szilard Nemeth
6cd540e964 YARN-7200. SLS generates a realtimetrack.json file but that file is missing the closing ']'. Contributed by Agshin Kazimli 2021-01-15 22:32:30 +01:00
Steve Loughran
724edb0354
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16830 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.

Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
2021-01-12 17:30:32 +00:00
Steve Loughran
05c9c2ed02 Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)"
This reverts commit d3014e01f3.
(fixing commit text before it is frozen)
2021-01-12 17:29:59 +00:00
Steve Loughran
d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16380 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.
2021-01-12 17:25:14 +00:00
Mehakmeet Singh
0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604)
Contributed by Mehakmeet Singh.
2021-01-12 15:48:09 +00:00
bilaharith
612330661b
HADOOP-17459. ADLS Gen1: Fixes for rename contract tests #2607
Contributed by Bilaharith
2021-01-12 14:00:48 +00:00
Sneha Vijayarajan
b612c310c2
HADOOP-17404. ABFS: Small write - Merge append and flush
- Contributed by Sneha Vijayarajan
2021-01-06 10:43:37 -08:00
bilaharith
d21c1c6576
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2551)
Contributed by bilaharith
2021-01-06 14:32:13 +00:00
Gabor Bota
42eb9ff68e
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
2021-01-05 15:43:01 +01:00
Ayush Saxena
77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
bilaharith
1448add08f
HADOOP-17347. ABFS: Read optimizations
- Contributed by Bilahari T H
2021-01-02 10:37:10 -08:00
Sneha Vijayarajan
5ca1ea89b3
HADOOP-17407. ABFS: Fix NPE on delete idempotency flow
- Contributed by Sneha Vijayarajan
2021-01-02 10:22:10 -08:00
Steve Loughran
617af28e80
HADOOP-17271. S3A connector to support IOStatistics. (#2580)
S3A connector to support the IOStatistics API of HADOOP-16830,

This is a major rework of the S3A Statistics collection to

* Embrace the IOStatistics APIs
* Move from direct references of S3AInstrumention statistics
  collectors to interface/implementation classes in new packages.
* Ubiquitous support of IOStatistics, including:
  S3AFileSystem, input and output streams, RemoteIterator instances
  provided in list calls.
* Adoption of new statistic names from hadoop-common

Regarding statistic collection, as well as all existing
statistics, the connector now records min/max/mean durations
of HTTP GET and HEAD requests, and those of LIST operations.

Contributed by Steve Loughran.
2020-12-31 21:55:39 +00:00
Sumangala
a35fc3871b
HADOOP-17422: ABFS: Set default ListMaxResults to max server limit (#2535)
Contributed by Sumangala Patki

TEST RESULTS:

namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
2020-12-21 06:40:36 +00:00