big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,998 Commits 2 Branches 2 Tags 264 MiB
09c8084191
Commit Graph

24998 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
xuzq
ee3ee98ee5 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473)
 2022-06-10 19:02:47 +00:00
Steve Loughran
de9f994338
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi (#4417) 
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.

Contributed by Steve Loughran

Change-Id: I61d087041561eeb8c9c42b5b7d8f0bb63f296b15
 2022-06-10 17:03:25 +01:00
Ashutosh Gupta
bdef321d52 HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
 2022-06-09 22:42:04 +09:00
slfan1989
a2f8a9e5d8 HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:59:34 +08:00
monthonk
7ec988d264
HADOOP-12020. Add s3a storage class option fs.s3a.create.storage.class (#3877) 
Adds a new option fs.s3a.create.storage.class which can
be used to set the storage class for files created in AWS S3.
Consult the documentation for details and instructions on how
disable the relevant tests when testing against third-party
stores.

Contributed by Monthon Klongklaew

Change-Id: I8cdebadf294a89fde08d98729ad96f251d58411c
 2022-06-08 20:02:07 +01:00
Viraj Jasani
516a2a8e44 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:56:07 +08:00
Viraj Jasani
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) (#4405) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-07 06:41:16 +08:00
Steve Loughran
03c2941d4b
HADOOP-18275. Update os-maven-plugin to 1.7.0 (#4397) 
Contributed by Steve Loughran

Change-Id: Ic4d442a37299dc8098b0bca3cc51beca6f058283
 2022-06-06 13:20:00 +01:00
Renukaprasad C
0c15daa77a
HDFS-16563. Namenode WebUI prints sensitive information on Token expiry (#4241) 
Contributed by Renukaprasad C

Change-Id: I5cd2cec1dd79917f810207821b3bdf4fe1a5d24c
 2022-06-06 11:08:57 +01:00
Samrat
7223a337f6 HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1d)
 2022-06-06 18:02:44 +09:00
Stephen O'Donnell
7d6b133af3 HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2022-06-01 20:54:56 +01:00
Ashutosh Gupta
de4c975710
HADOOP-18238. Fix reentrancy check in SFTPFileSystem.close() (#4330) 
Contributed by Ashutosh Gupta

Change-Id: I2742675add74259a93b3762a80c7ab5ee6d08c37
 2022-05-30 17:34:45 +01:00
GuoPhilipse
dd9b8af9c4
HADOOP-18269. Misleading method name in DistCpOptions.(#4216) 
Contributed by guophilipse

Change-Id: I5e75d030406997339c20e970483825e529d9cd10
 2022-05-30 14:04:33 +01:00
slfan1989
91f19bf8fa
HADOOP-18244. Fix Hadoop-Common JavaDoc Error on branch-3.3 (#4327). Contributed by fanshilun. 2022-05-29 11:31:16 +05:30
Ashutosh Gupta
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785)
 2022-05-26 15:28:27 -07:00
Stephen O'Donnell
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a)
 2022-05-26 10:14:50 -07:00
Wei-Chiu Chuang
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) (#4304) 
(cherry picked from commit cee8c62498)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java

(cherry picked from commit dd79aee635fdc61648e0c87bea1560dc35aee053)

Co-authored-by: caozhiqiang <lfxy@163.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-27 00:50:40 +08:00
Ashutosh Gupta
e0732baeb8 YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* (#4271) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e3e9369c1d)
 2022-05-26 15:58:47 +09:00
Masatake Iwasaki
241fb6b2a7 HADOOP-18251. Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py. (#4344) 
(cherry picked from commit 6b331dde31)
 2022-05-26 03:28:55 +00:00
Michael Stack
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4347) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 17:02:28 -07:00
Ashutosh Gupta
8c492a1d65 HADOOP-18240. Upgrade Yetus to 0.14.0 (#4328) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 84b0455cf8)
 2022-05-25 17:32:19 +09:00
jianghuazhu
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. (#4342). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-24 23:47:45 +08:00
Owen O'Malley
1f111d6a41 YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 
(cherry picked from commit f390edaec4)
 2022-05-24 05:17:34 +00:00
Viraj Jasani
ab3a9cedc9 HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:47:18 +08:00
Ashutosh Gupta
57fe613299
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb910bd906)

 Conflicts:
	hadoop-project/pom.xml
 2022-05-21 03:17:15 +09:00
Szilard Nemeth
90ec4418c7 YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-18 14:34:08 +02:00
Szilard Nemeth
4f112e3138 YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-18 14:25:35 +02:00
Szilard Nemeth
b4550b3356 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-18 14:08:41 +02:00
Ashutosh Gupta
b0012ee578 HADOOP-18237. Upgrade Apache Xerces Java to 2.12.2 (#4318) 
Upgrade Apache Xerces Java to 2.12.2 due to handle vulnerability CVE-2022-23437

Contributed by Ashutosh Gupta
 2022-05-17 20:37:30 +01:00
Zilong Zhu
2dd5852ad0 YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280) 
(cherry picked from commit 79cc0caf9f)
 2022-05-18 03:03:13 +09:00
Steve Loughran
fe306ce57e HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:09 +01:00
Ashutosh Gupta
c2071947f8
HADOOP-18234. Fix s3a access point xml examples (#4309) 
Contributed by Ashutosh Gupta

Change-Id: Ib99a4c7113d60690c1301d80cd5c27a43c463db0
 2022-05-16 17:48:04 +01:00
Ashutosh Gupta
a50ce18f4d YARN-11092. Upgrade jquery ui to 1.13.1 (#4260) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 931abbd927)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java
 2022-05-16 23:26:46 +09:00
Michael Stack
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) (#4246) 
Cherry-pick backport of 9ed8d60511
 2022-05-15 21:32:48 -07:00
Ashutosh Gupta
d58f9d438f
YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6985f9aabe)
 2022-05-14 01:41:31 +09:00
Viraj Jasani
e16ae55833 HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 09:00:31 +08:00
daimin
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4296) 2022-05-11 07:56:53 +08:00
hchaverr
1e043b937a
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times 
Fixes #4266

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-10 14:52:31 -07:00
Ashutosh Gupta
277daca91f
HADOOP-17479. Fix the examples of hadoop config prefix (#4197) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 40a8b9a6a5)
 2022-05-08 08:09:47 +09:00
Ashutosh Gupta
c879230547
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9)
 2022-05-08 08:06:14 +09:00
Ashutosh Gupta
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5b)
 2022-05-08 07:39:17 +09:00
daimin
d711d200fe
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29401c8203)
 2022-05-06 10:35:55 -07:00
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Daniel Carl Jones
4c977f5f71 HADOOP-18168. Fix S3A ITestMarkerTool use of purged public bucket. (#4140) 
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones

Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
 2022-05-03 14:28:46 +01:00
Steve Loughran
c2b2494d8b
HADOOP-18198. add -mvnargs option to create-release command line (#4239) 
This allows for builds to be run with options like
--mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false"

Contributed by Ayush Saxena.

Change-Id: I0e3d421da13bf0bc32cbbfbfb2c428828bdbb84c
 2022-05-03 14:07:15 +01:00
jteagles
f155abc572
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d4a91bd0c0)
 2022-05-03 03:15:10 +09:00
Ashutosh Gupta
4f9e607126
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a74acc755e)
 2022-05-03 03:10:43 +09:00
Viraj Jasani
ee450bbbc6
HDFS-16528. Reconfigure slow peer enable for Namenode (#4251) 2022-05-02 08:03:02 +08:00
Steve Loughran
cc204c9611
HADOOP-16202. Enhanced openFile(): hadoop-azure changes. (#2584/4) 
Stops the abfs connector warning if openFile().withFileStatus()
is invoked with a FileStatus is not an abfs VersionedFileStatus.

Contributed by Steve Loughran.

Change-Id: I85076b365eb30aaef2ed35139fa8714efd4d048e
 2022-04-27 19:24:33 +01:00
Steve Loughran
9db61adeda
HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3) 
S3A input stream support for the few fs.option.openfile settings.
As well as supporting the read policy option and values,
if the file length is declared in fs.option.openfile.length
then no HEAD request will be issued when opening a file.
This can cut a few tens of milliseconds off the operation.

The patch adds a new openfile parameter/FS configuration option
fs.s3a.input.async.drain.threshold (default: 16000).
It declares the number of bytes remaining in the http input stream
above which any operation to read and discard the rest of the stream,
"draining", is executed asynchronously.
This asynchronous draining offers some performance benefit on seek-heavy
file IO.

Contributed by Steve Loughran.

Change-Id: I9b0626bbe635e9fd97ac0f463f5e7167e0111e39
 2022-04-27 19:23:56 +01:00