big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,023 Commits 2 Branches 2 Tags
Commit Graph

8546 Commits

Author SHA1 Message Date
Steve Vaughan
833fc64558
HDFS-16684. Exclude the current JournalNode (#4786) 
The JournalNodeSyncer will include the local instance in syncing when using a bind host (e.g. 0.0.0.0).  There is a mechanism that is supposed to exclude the local instance, but it doesn't recognize the meta-address as a local address.

Running with bind addresses set to 0.0.0.0, the JournalNodeSyncer will log attempts to sync with itself as part of the normal syncing rotation.  For an HA configuration running 3 JournalNodes, the "other" list used by the JournalNodeSyncer will include 3 proxies.

Exclude bound local addresses, including the use of a wildcard address in the bound host configurations, while still allowing multiple instances on the same host.

Allow sync attempts with unresolved addresses, so that sync attempts can drive resolution as servers become available.

Backport.
Signed-off-by: stack <stack@apache.org>
 2022-08-28 11:15:04 -07:00
zhengchenyu
3edddaf9fc HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 231a4468cdb83f9c2ff8897e70fe7c3d23b58cf4)
 2022-08-25 10:41:04 -07:00
Steve Vaughan
4138661010
HDFS-16625. Check assumption about PMDK availability (#4788) 
Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
 2022-08-23 19:35:59 +09:00
Steve Loughran
9c5228cf6b
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.

Change-Id: I98b0fed54da3b810c3f23fe5b12e673937916257
 2022-08-05 14:02:28 +01:00
xuzq
e024d1a3f8
HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-03 13:02:42 +05:30
Masatake Iwasaki
ff13f9ee8b Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57dfd41385dbce5de90620c48a45abb263)
 2022-07-22 02:31:34 +00:00
Viraj Jasani
4ba463069b
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4485) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-23 17:30:01 +08:00
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482) 
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
 2022-06-22 13:09:50 +01:00
Viraj Jasani
7561dbd134
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:44:30 +05:30
KevinWikant
33ab84f2e2 HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6028f9604421a9ae10dda34bc1f9532)
 2022-06-20 11:24:45 +09:00
jianghuazhu
18a5e843bc HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:20:34 +08:00
xuzq
ee3ee98ee5 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a47311bad542964c42c1f776e4350446b9)
 2022-06-10 19:02:47 +00:00
Ashutosh Gupta
bdef321d52 HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c941703d001d216733a01569a5bc83c348c9)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
 2022-06-09 22:42:04 +09:00
slfan1989
a2f8a9e5d8 HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:59:34 +08:00
Viraj Jasani
516a2a8e44 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:56:07 +08:00
Viraj Jasani
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) (#4405) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-07 06:41:16 +08:00
Samrat
7223a337f6 HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1dd844b13559256f16950e32093938db)
 2022-06-06 18:02:44 +09:00
Stephen O'Donnell
7d6b133af3 HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90ef89b633c9b5c13a79aa1ac11c92eb5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2022-06-01 20:54:56 +01:00
Ashutosh Gupta
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785c012794a2dcf122dc8717b18beb0f0)
 2022-05-26 15:28:27 -07:00
Stephen O'Donnell
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a48468b56fd45c86412569cd927f749)
 2022-05-26 10:14:50 -07:00
Wei-Chiu Chuang
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) (#4304) 
(cherry picked from commit cee8c62498f55794f911ce62edfd4be9e88a7361)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java

(cherry picked from commit dd79aee635fdc61648e0c87bea1560dc35aee053)

Co-authored-by: caozhiqiang <lfxy@163.com>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-27 00:50:40 +08:00
Michael Stack
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4347) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 17:02:28 -07:00
jianghuazhu
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. (#4342). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-24 23:47:45 +08:00
Viraj Jasani
ab3a9cedc9 HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:47:18 +08:00
Steve Loughran
fe306ce57e HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:09 +01:00
Michael Stack
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) (#4246) 
Cherry-pick backport of 9ed8d60511dccf96108239c5c96e108a7d4bc975
 2022-05-15 21:32:48 -07:00
Viraj Jasani
e16ae55833 HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 09:00:31 +08:00
daimin
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4296) 2022-05-11 07:56:53 +08:00
Ashutosh Gupta
c879230547
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9789eb5b386729f52b7bb9e52fa5352)
 2022-05-08 08:06:14 +09:00
Ashutosh Gupta
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5beac5e1f2da5907162d9438e98b98af)
 2022-05-08 07:39:17 +09:00
daimin
d711d200fe
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29401c820377d02a992eecde51083cf87f8e57af)
 2022-05-06 10:35:55 -07:00
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Viraj Jasani
ee450bbbc6
HDFS-16528. Reconfigure slow peer enable for Namenode (#4251) 2022-05-02 08:03:02 +08:00
Viraj Jasani
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-27 10:30:07 +01:00
daimin
b62a460fd9 HDFS-16519. Add throttler to EC reconstruction (#4101) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit aebd55f7883c6b12afe5faeb776ab0e0b83420da)
 2022-04-23 12:47:34 +09:00
litao
9132eeb4dd HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc48023c0c71dd428df40a4352feede373b)
 2022-04-23 12:15:52 +09:00
S O'Donnell
5e137ac33e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 8ae033d1a3d65281d3ad1f07f430e74d3a01b1b9.
 2022-04-20 20:45:17 +01:00
qinyuren
2ff91232bc HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd173749f2af4f17946fb37c4c72e2de26764)
 2022-04-20 15:07:38 +09:00
Stephen O'Donnell
8ae033d1a3 HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee03639f41a022dd07d5fc04e3aa65a94b5f)
 2022-04-19 11:12:36 +01:00
qinyuren
c913dc3072 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0d5f522f3b799ea72c6c887d5d2c42d)
 2022-04-19 13:38:58 +09:00
jianghuazhu
cfe2d8aa79
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e7120e7af250ebf0b4bafb40b412d3e11c)
 2022-04-19 11:13:40 +08:00
Ashutosh Gupta
1eb4f9ef04
HDFS-16536. TestOfflineImageViewer fails on branch-3.3 (#4182) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-18 16:05:42 +09:00
Quanlong Huang
9ae903dd1b
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
(cherry picked from commit 35d4c02bccd303b5a2608a952fa1f8487001087b)
 2022-04-18 10:35:08 +08:00
daimin
0ef1a13f01
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 
(cherry picked from commit c65c383b7ebef48c638607f15ba35d61554982cb)
 2022-04-14 16:16:36 +08:00
Takanobu Asanuma
52abc9f132 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959ca0a68c52ed6a9c721704e57afbcc7)
 2022-04-14 11:50:42 +09:00
qinyuren
07dface36a HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a112334e48087bd60674538af739922a)
 2022-04-13 11:48:19 +09:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development 
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
 2022-04-12 14:09:08 +01:00
singer-bin
26705bbc60
HDFS-16457. Make fs.getspaceused.classname reconfigurable (apache#4069) (#4156) 2022-04-11 14:59:34 +09:00
Takanobu Asanuma
30afe7ca20 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 
(cherry picked from commit 37650ced81964efe7a9ea45bf505af96b8035d53)
 2022-04-08 18:39:26 +09:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-07 08:33:13 +09:00