Commit Graph

332 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
8af2d2feb2
Update version to 3.3.6 2023-06-12 15:34:41 -07:00
Wei-Chiu Chuang
dea53dad67
HADOOP-18761. Remove mysql-connector-java (#5731)
(cherry picked from commit e1bb4acd36)
2023-06-12 15:33:25 -07:00
Dongjoon Hyun
20d073cb2c
HADOOP-18718. Fix several maven build warnings (#5592). Contributed by Dongjoon Hyun.
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit fb16e00da0)

Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
2023-06-12 10:40:41 -07:00
hfutatzhanghb
c2385c021b
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) (#5423). Contributed by farmmamba
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-23 09:10:29 +05:30
Wei-Chiu Chuang
99312bdfdb
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) (#5619)
* HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553)

The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu

(cherry picked from commit 0e46388474)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestViewDistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithAcl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRetryCacheMetrics.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFSImageWithOrderedSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestOrderedSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotDeletion.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerForErasureCodingPolicy.java

Co-authored-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
2023-05-09 06:20:56 +08:00
Owen O'Malley
c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira
5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu
9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-02-24 13:31:53 -08:00
hchaverr
eab7215354
HADOOP-18535. Implement token storage solution based on MySQL
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2023-02-22 14:02:13 -08:00
huhaiyang
de08baded6
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301)
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
2023-02-09 13:29:47 +00:00
Simbarashe Dzinamarira
dde22d522c HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 11:47:57 -08:00
Owen O'Malley
6b23e70539 HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 14:19:19 -08:00
Simbarashe Dzinamarira
9d37ee082c HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-28 16:54:25 -08:00
Owen O'Malley
8ff54dac58 HDFS-16844: RBF: Adds resilancy when StateStore gets exceptions. (#5138)
Allows the StateStore to stay up when there are errors reading the data.
2022-11-18 09:29:08 -08:00
Melissa You
02aedd7811
Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 (#5112)
* HDFS-15383. RBF: Add support for router delegation token without watch (#2047)

Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.

* HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: lfengnan <lfengnan@uber.com>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
2022-11-07 13:29:50 -08:00
Xing Lin
760144f135
HDFS-16628. RBF: Correct target directory when move to trash for kerberos login user. (#4974)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-10-11 16:14:39 +09:00
Xing Lin
7d7f7a9e9b
HDFS-16024. RBF: Rename data to the Trash should be based on src location (#4962)
(cherry picked from commit e18d806212)

Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-10-10 00:33:48 +09:00
Simba Dzinamarira
e28dc524f6
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-08-23 11:27:21 -07:00
Steve Vaughan
a73efb2d55
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) (#4790) 2022-08-22 20:26:03 -07:00
jianghuazhu
2123859d60
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f176d080c)
2022-08-21 07:31:01 +09:00
Ayush Saxena
df4e59318f
HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun.
Only Prod Changes: Test already cherry-picked as part of HDFS-16310 via (496657c63f)
2022-07-23 23:48:02 +05:30
Steve Loughran
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 (#4482)
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.

Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.

Contributed by Steve Loughran
2022-06-22 13:09:50 +01:00
Ashutosh Gupta
bdef321d52 HDFS-16576. Remove unused imports in HDFS project (#4389)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java
2022-06-09 22:42:04 +09:00
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Steve Loughran
44e662272f
HADOOP-18198. Preparing for 3.3.4 development
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
2022-04-12 14:09:08 +01:00
Masatake Iwasaki
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052)
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-04-07 08:33:13 +09:00
Owen O'Malley
1175acbb75 HDFS-13248: Namenode needs to use the actual client IP when going through the
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
2022-03-21 10:24:54 -07:00
Owen O'Malley
c52b97d084 HDFS-16495: RBF should prepend the client ip rather than append it.
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-03-14 10:30:29 -07:00
litao
496657c63f HDFS-16310. RBF: Add client port to CallerContext for Router (#3635)
Cherry-picked from 5b05068f by Owen O'Malley
2022-03-14 10:29:30 -07:00
Hui Fei
2479d4ab6c HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang
Cherry-picked from 264c948e by Owen O'Malley
2022-03-14 10:29:04 -07:00
Hui Fei
8e129e5b8d HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363)
Cherry-picked from 518a212c by Owen O'Malley
2022-03-14 10:28:55 -07:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791)
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
2022-02-11 23:18:25 +08:00
wangzhaohui
cc7b7e174c HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820)
(cherry picked from commit 8d251bd629)
2021-12-21 11:46:23 +00:00
Chao Sun
e079fa6577 Preparing for 3.3.3 development 2021-11-16 16:02:34 -08:00
Ayush Saxena
5b1cf17f88
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:53 +05:30
Symious
c0f32f3cf8 HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421)
(cherry picked from commit 7f6553af75)
2021-09-13 10:48:09 +08:00
Symious
8affaa6312 HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379)
(cherry picked from commit c0890e6d04)
2021-09-10 10:01:27 +08:00
Viraj Jasani
b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115)
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
2021-06-21 10:28:05 +09:00
Viraj Jasani
ef3a2a3b3b HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:23:52 +05:30
litao
02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a)
2021-06-08 16:59:59 +09:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)"
This reverts commit 4ffe5eb1dd.
2021-05-24 17:37:18 +08:00
Wei-Chiu Chuang
fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
2021-05-17 18:57:46 -07:00
Aryan Gupta
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
2021-05-13 12:16:17 +08:00
Akira Ajisaka
25244c4eec
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970)
(cherry picked from commit 7c561e2a02)
2021-05-04 00:26:34 +09:00
lfengnan
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910)
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c)
2021-05-02 19:19:55 +09:00
Wei-Chiu Chuang
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945)
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029)
2021-05-02 11:12:37 +09:00
litao
0e9042fed5 HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit b968fa0957)
2021-04-28 18:52:05 +09:00
Ayush Saxena
f809b863fe HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.
(cherry picked from commit 6e416a83d1)
2021-04-28 18:51:43 +09:00
Ayush Saxena
3c7433f2a1 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
(cherry picked from commit 8d49229c37)
2021-04-28 18:51:37 +09:00