9329 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
7db9895000 Post release update
* Add jdiff xml files from 3.3.6 release.
* Declare 3.3.6 as the latest stable release.
* Copy release notes.
2023-06-26 15:59:39 +00:00
Wei-Chiu Chuang
8af2d2feb2
Update version to 3.3.6 2023-06-12 15:34:41 -07:00
Wei-Chiu Chuang
dea53dad67
HADOOP-18761. Remove mysql-connector-java (#5731)
(cherry picked from commit e1bb4acd3686720da1a50676400f080cc9403922)
2023-06-12 15:33:25 -07:00
Dongjoon Hyun
20d073cb2c
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun.
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit fb16e00da0e7c50cba8bf238fad7d09281717848)

Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
2023-06-12 10:40:41 -07:00
hfutatzhanghb
0fd2b10bdc
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat. (#5408). Contributed by ZhangHB.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-06-12 13:42:39 +08:00
hfutatzhanghb
a804f37ed5
HDFS-17003. Erasure Coding: invalidate wrong block after reporting bad blocks from datanode (#5643). Contributed by hfutatzhanghb.
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 0e6bd09ae3c79d5869ffd9ab6ad579786b1f3cd7)
2023-06-08 18:22:57 +08:00
hfutatzhanghb
c2385c021b
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) (#5423). Contributed by farmmamba
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-23 09:10:29 +05:30
huhaiyang
19a6762639
HDFS-17017. Fix the issue of arguments number limit in report command in DFSAdmin (#5667). Contributed by Haiyang Hu.
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-21 09:06:53 +05:30
NishthaShah
adb673fdaf
HDFS-17022. Fix the exception message to print the Identifier pattern (#5678). Contributed by Nishtha Shah.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-20 14:41:49 +05:30
wangzhaohui
5cafa1fdb9
HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-18 12:13:56 +05:30
Wei-Chiu Chuang
99312bdfdb
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) (#5619)
* HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553)

The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu

(cherry picked from commit 0e46388474f52cd38578d946c26ba762be05ef1f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestViewDistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithAcl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRetryCacheMetrics.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFSImageWithOrderedSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestOrderedSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerForErasureCodingPolicy.java

Co-authored-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
2023-05-09 06:20:56 +08:00
Ayush Saxena
a226016c52
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena.
Reviewed-by: Steve Loughran <stevel@apache.org>
2023-05-02 20:12:22 +05:30
fanluoo
408c5c53b1
HDFS-16897. Fix abundant Broken pipe exception in BlockSender (#5329). Contributed by fanluo.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-29 17:29:55 +05:30
cxzl25
4d052a2456
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
2023-04-26 02:12:40 +05:30
Sebastian Baunsgaard
919c3f615b
HADOOP-18660. Filesystem Spelling Mistake (#5475).
Contributed by Sebastian Baunsgaard.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-25 19:59:54 +01:00
wangzhaohui
1ea5fbd192
HDFS-16988. Improve NameServices info at JournalNode web UI (#5584). Contributed by Zhaohui Wang.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-24 01:14:18 +05:30
Steve Loughran
a505940a2f
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558)
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
2023-04-18 10:12:41 +01:00
Stephen O'Donnell
c39696ab1d HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/
(cherry picked from commit eee2ea075d10b197837ceca8cbb2260b48b29d10)
2023-03-15 10:17:55 +00:00
Stephen O'Donnell
d7b89d08e1 HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460)
(cherry picked from commit ca6f5afb6da7f1c40ba42061d34b60b54bb01712)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java
2023-03-11 17:04:33 +00:00
zhangshuyan
8cc57f5eb1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450 #5471). Contributed by Shuyan Zhang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-03-11 16:09:06 +08:00
slfan1989
e12bc4e1d8
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434)
Contributed by Shilun Fan
2023-03-06 15:45:42 +00:00
Tom
703158c9c6
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) (#5444)
Cherry picked from: 162288bc0af944f116a4dd73e27f4676a204d9e9

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
2023-03-02 00:44:30 +00:00
ZanderXu
a7c1fad0c9 HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400)
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 6bd24448154fcd3ab9099d7783cc7f7f76c61e08)
2023-03-01 16:20:50 -08:00
Viraj Jasani
91ce13ea01
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432)
Contributed by Viraj Jasani
2023-03-01 18:53:52 +00:00
rdingankar
94b3c6dd90 HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397)
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
2023-02-27 15:49:26 -08:00
Owen O'Malley
c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira
5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu
9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-02-24 13:31:53 -08:00
hchaverr
eab7215354
HADOOP-18535. Implement token storage solution based on MySQL
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2023-02-22 14:02:13 -08:00
Viraj Jasani
e47ff5f3c0
HDFS-16925. Namenode audit log to only include IP address of client (#5413)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-02-21 08:33:54 +08:00
Steve Vaughan
d437571fe2
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372)
This is a followup to the original patch, 08f58ecf07f, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
2023-02-13 14:47:00 +00:00
Viraj Jasani
5099077f4d
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) (#5358)
* ADDENDUM Remove unused variables from testDataNodeMXBeanLastHeartbeats (#5373)

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-02-12 11:53:59 +05:30
Ayush Saxena
b6d732b96b
HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2023-02-12 01:57:34 +05:30
Masatake Iwasaki
d85c12ee69
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2023-02-12 01:50:39 +05:30
Ahmed Hussein
c17734b747
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2023-02-12 01:50:26 +05:30
huhaiyang
de08baded6
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301)
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
2023-02-09 13:29:47 +00:00
Steve Vaughan
b6680907f9
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342)
Contributed by Steve Vaughan Jr
2023-02-09 13:29:00 +00:00
Tao Li
10b42e9306 HDFS-16902. Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334)
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-02-03 10:02:45 +08:00
huhaiyang
80b42625cf HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296)
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 88c8ac750dc630040a37014dd13391922e625292)
2023-01-31 17:52:29 +09:00
Viraj Jasani
63443be5f4 HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300)
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 04f3573f6ad07c64c1e2cf8a82fb3e9263366541)
2023-01-18 21:14:17 +00:00
Viraj Jasani
c5cf845d78 HDFS-16887 Log start and end of phase/step in startup progress (#5292)
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 1263e024b9744574bf4042b2f479a749c340fbcd)
2023-01-12 22:27:35 +00:00
Simbarashe Dzinamarira
dde22d522c HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 11:47:57 -08:00
Chengbing Liu
af96e0f5b3 HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246)
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 4cf304de4520bac5be265501fdb056114b4154f5)
2023-01-10 10:04:05 -08:00
ZanderXu
a49378cb93 HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872)
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>

(cherry picked from commit 8d221255f2daf2c511bc878598bdefb8b2342e87)
2023-01-10 10:03:45 -08:00
Xing Lin
d43fa95043 HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down (#5160)
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit f7bdf6c667d503c0eebbc3efcdfe4ba6bf2d6275)
2022-12-16 08:47:33 -08:00
dingshun3016
ebd2407d48 HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050)
(cherry picked from commit 02afb9ebe137a024a3dae49af3bf03dacb8c5fc8)
2022-12-05 16:39:49 +09:00
Ashutosh Gupta
fa2a0a603a HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452)
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
(cherry picked from commit b7edc6c60ca9e17c4f6a7d07aad5efbd7c2e3b40)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java
2022-12-01 15:00:32 +09:00
Kidd5368
8c7f2ddc10
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128)
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
(cherry picked from commit 72749a4ff8a240867a822bb9ae1382d3235083ea)
2022-11-29 17:45:12 -08:00
Owen O'Malley
6b23e70539 HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 14:19:19 -08:00
Simbarashe Dzinamarira
9d37ee082c HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-28 16:54:25 -08:00