big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,215 Commits 2 Branches 2 Tags 264 MiB
bb08c90228
Commit Graph

25215 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
bb08c90228
HADOOP-18416. fix ITestS3AIOStatisticsContext test failure (#4931) 
Uncomment the S3ATestUtils-side part of the original patch.
 2023-04-28 12:03:30 +01:00
Viraj Jasani
0fd36df1d2
HADOOP-18377. hadoop-aws build to add a -prefetch profile to run all tests with prefetching (#4914) 
Contributed by Viraj Jasani
 2023-04-28 12:03:30 +01:00
Viraj Jasani
76e243aacb
HADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field (#4926) 
Follow-on to HADOOP-18455.

Contributed by Viraj Jasani
 2023-04-28 12:03:30 +01:00
Viraj Jasani
f07be3bec2
HADOOP-18455. S3A prefetching executor should be closed (#4879) 
follow-on patch to HADOOP-18186. 

Contributed by: Viraj Jasani
 2023-04-28 12:03:30 +01:00
Viraj Jasani
1c2c6785a0
HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work (#4796) 
Contributed by Viraj Jasani
 2023-04-28 12:03:30 +01:00
Viraj Jasani
f00d77fda4
HADOOP-18380. fs.s3a.prefetch.block.size to be read through longBytesOption (#4762) 
Contributed by Viraj Jasani.
 2023-04-28 12:03:30 +01:00
Steve Loughran
4ce763a322
HADOOP-18028. High performance S3A input stream (#4752) 
This is the the preview release of the HADOOP-18028 S3A performance input stream.
It is still stabilizing, but ready to test.

Contains

HADOOP-18028. High performance S3A input stream (#4109)
	Contributed by Bhalchandra Pandit.

HADOOP-18180. Replace use of twitter util-core with java futures (#4115)
	Contributed by PJ Fanning.

HADOOP-18177. Document prefetching architecture. (#4205)
	Contributed by Ahmar Suhail

HADOOP-18175. fix test failures with prefetching s3a input stream (#4212)
 Contributed by Monthon Klongklaew

HADOOP-18231.  S3A prefetching: fix failing tests & drain stream async.  (#4386)

	* adds in new test for prefetching input stream
	* creates streamStats before opening stream
	* updates numBlocks calculation method
	* fixes ITestS3AOpenCost.testOpenFileLongerLength
	* drains stream async
	* fixes failing unit test

	Contributed by Ahmar Suhail

HADOOP-18254. Disable S3A prefetching by default. (#4469)
	Contributed by Ahmar Suhail

HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458)

	This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with
	new statistic names in StreamStatistics.

	This stream is not (yet) IOStatisticsContext aware.

	Contributed by Ahmar Suhail

HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk
HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums.
HADOOP-18318. Update class names to be clear they belong to S3A prefetching
	Contributed by Steve Loughran
 2023-04-28 12:03:29 +01:00
PJ Fanning
040c23c768
HADOOP-18712. Upgrade to jetty 9.4.51 due to cve. Contributed by PJ Fanning. (#5574) (#5585) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-26 18:51:04 +05:30
cxzl25
4d052a2456
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-26 02:12:40 +05:30
Sebastian Baunsgaard
919c3f615b
HADOOP-18660. Filesystem Spelling Mistake (#5475). 
Contributed by Sebastian Baunsgaard.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 19:59:54 +01:00
Steve Loughran
0b56be3ca4
MAPREDUCE-7437. MR Fetcher class to use an AtomicInteger to generate IDs. (#5579) 
...as until now it wasn't thread safe

Contributed by Steve Loughran
 2023-04-25 19:56:18 +01:00
Ayush Saxena
d7d36b9d2a
HADOOP-18689. Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project (#5502) (#5586) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-25 21:26:59 +05:30
Steve Loughran
21cf507db3
HADOOP-17450. Add Public IOStatistics API -missed backport (#5590) 
This cherrypicks SemaphoredDelegatingExecutor HADOOP-17450 changes
from trunk somehow they didn't get into the main IOStatistics backport
to branch-3.3
 2023-04-25 15:02:56 +01:00
Tamas Domok
1b59e3123b
HADOOP-18705. ABFS should exclude incompatible credential providers. (#5560) 
Contributed by Tamas Domok.
 2023-04-24 15:48:02 +01:00
Doroszlai, Attila
13d3cfd311
HADOOP-18714. Wrong StringUtils.join() called in AbstractContractRootDirectoryTest (#5588) 
(cherry picked from commit 5b23224970)
 2023-04-24 15:49:20 +02:00
wangzhaohui
1ea5fbd192
HDFS-16988. Improve NameServices info at JournalNode web UI (#5584). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:14:18 +05:30
Nikita Eshkeev
7a32e7cc38
HADOOP-18597. Simplify single node instructions for creating directories for Map Reduce. (#5305) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:14:09 +05:30
Christos Bisias
57ff8bdb67 HADOOP-18691. Add a CallerContext getter on the Schedulable interface (#5540) 2023-04-20 10:13:33 -07:00
Steve Loughran
a505940a2f
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:41 +01:00
Steve Loughran
f5464831a0
HADOOP-18696. ITestS3ABucketExistence arn test failures. (#5557) 
Explicitly sets the fs.s3a.endpoint.region to eu-west-1 so
the ARN-referenced fs creation fails with unknown store
rather than IllegalArgumentException.

Steve Loughran
 2023-04-17 10:21:01 +01:00
dependabot[bot]
8e15216045 HADOOP-18693. Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project (#5427) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-16 06:51:17 -07:00
Dongjoon Hyun
854eddb0cc
HADOOP-18590. Publish SBOM artifacts (#5555). Contributed by Dongjoon Hyun. 2023-04-15 21:40:25 +05:30
sreeb-msft
f324efd247
HADOOP-18012. ABFS: Enable config controlled ETag check for Rename idempotency (#5488) 
To support recovery of network failures during rename, the abfs client
fetches the etag of the source file, and when recovering from a
failure, uses this tag to determine whether the rename succeeded
before the failure happened.

* This works for files, but not directories
* It adds the overhead of a HEAD request before each rename.
* The option can be disabled by setting "fs.azure.enable.rename.resilience"
  to false

Contributed by Sree Bhattacharyya
 2023-04-05 15:07:39 +01:00
HarshitGupta11
42ed2b9075
HADOOP-18684. S3A filesystem to support binding to to other URI schemes (#5521) 
Contributed by Harshit Gupta
 2023-04-05 14:57:27 +01:00
Chris Nauroth
3400e8257e HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3. 
Closes #5511

Signed-off-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-03 22:58:53 +00:00
Pranav Saxena
054afa1180
HADOOP-18647. x-ms-client-request-id to identify the retry of an API. (#5437) 
The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous
operation

Contributed by Pranav Saxena
 2023-03-30 14:26:12 +01:00
Viraj Jasani
20d3b9cc46
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) (#5374) 2023-03-30 07:13:10 +08:00
Anmol Asrani
6306f5b2bc
HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 
This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
 2023-03-28 16:32:01 +01:00
Pranav Saxena
2b156c2b32
HADOOP-18606. ABFS: Add reason in x-ms-client-request-id on a retried API call. (#5299) 
Contributed by Pranav Saxena
 2023-03-28 12:00:57 +01:00
Masatake Iwasaki
dd9ef9e0e7
HADOOP-17746. Compatibility table in directory_markers.md doesn't render right. (#3116) 
Contributed by Masatake Iwasaki
 2023-03-15 17:11:30 +00:00
Stephen O'Donnell
c39696ab1d HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/ 
(cherry picked from commit eee2ea075d)
 2023-03-15 10:17:55 +00:00
Steve Loughran
0dd4e500b0
HADOOP-18661. Fix bin/hadoop usage script terminology. (#5473) 
Followup to HADOOP-13209: s/slaves/r/workers in
the usage message you get when you type "bin/hadoop"

Contributed by Steve Loughran
 2023-03-13 12:24:10 +00:00
Stephen O'Donnell
d7b89d08e1 HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460) 
(cherry picked from commit ca6f5afb6d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java
 2023-03-11 17:04:33 +00:00
zhangshuyan
8cc57f5eb1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450 #5471). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-11 16:09:06 +08:00
slfan1989
4a23389f5e
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-03-10 15:16:28 +00:00
Steve Loughran
b75ced1e5d
HADOOP-17836. Improve logging on ABFS error reporting (#3281) 
Contributed by Steve Loughran.
 2023-03-08 15:31:16 +00:00
slfan1989
e12bc4e1d8
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434) 
Contributed by Shilun Fan
 2023-03-06 15:45:42 +00:00
Varun Saxena
157af0cb22 YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary

(cherry picked from commit 2a0dc2ab2f)
 2023-03-05 23:02:17 +05:30
Tom
703158c9c6
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) (#5444) 
Cherry picked from: 162288bc0a

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-02 00:44:30 +00:00
ZanderXu
a7c1fad0c9 HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 6bd2444815)
 2023-03-01 16:20:50 -08:00
Viraj Jasani
91ce13ea01
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:52 +00:00
Steve Loughran
bca38f84af
HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429) 
POM and LICENSE fixup of transient dependencies
* Exclude hadoop-cloud-storage imports which come in with hadoop-common
* Add explicit import of hadoop's org.codehaus.jettison declaration
  to hadoop-aliyun
* Tune aliyun jars imports
* Cut duplicate and inconsistent hbase-server declarations from
  hadoop-project
* Update LICENSE-binary for the current set of libraries in the
  hadoop 3.3.5 release.

Contributed by Steve Loughran
 2023-02-28 14:05:13 +00:00
rdingankar
94b3c6dd90 HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 15:49:26 -08:00
Owen O'Malley
c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira
5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu
9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-24 13:31:53 -08:00
Steve Loughran
35e04ff52a
HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file 
+ add a link to mukund's apachecon talk

Change-Id: I3d04b385ff1312aabf2a81d034f54f124d544a54
 2023-02-23 13:26:54 +00:00
Steve Loughran
92d6826d10
Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" 
Causes HADOOP-18641. cyclonedx maven plugin breaks on recent maven releases

This reverts branch-3.3 commit b6fd322307.
 2023-02-23 11:25:50 +00:00
hchaverr
eab7215354
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 14:02:13 -08:00
Ayush Saxena
84e999b35c
HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp. (#5409) 
Followup to the original HADOOP-18582.

Temporary path cleanup is re-enabled for -append jobs
as these will create temporary files when creating or overwriting files.

Contributed by Ayush Saxena
 2023-02-22 19:32:05 +00:00