big-data/hadoop
big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
27,203 Commits 2 Branches 2 Tags
Commit Graph

337 Commits

Author SHA1 Message Date
Takanobu Asanuma
c4f7a3625b
HDFS-17435. Fix TestRouterRpc failed (#6650) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-03-26 09:38:35 +09:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up (#6498) 2024-02-13 16:24:23 -08:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider (#6510) 
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
 2024-02-13 10:49:39 +09:00
Alexander
20d8596af2
HDFS-17353. Fix failing RBF module tests. (#6491) Contributed by Alexander Bogdanov 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-02-03 19:26:30 +08:00
LiuGuH
8193a84020
HDFS-17324. RBF: Router should not return nameservices that not enable observer r… (#6412) 2024-01-23 12:30:08 -08:00
LiuGuH
2a1ee8dfcd
HDFS-17311. RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue. (#6392) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-01-20 07:55:23 +08:00
Jian Zhang
1036544480
HDFS-17302. RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation. (#6380) 2024-01-19 14:02:21 -08:00
LiuGuH
7d3b6a36b8
HDFS-17306. RBF: Router should not return nameservices that does not enable observer nodes in RpcResponseHeaderProto (#6385) 2024-01-04 14:43:11 -08:00
LiuGuH
b4fed58c6a
HDFS-17285. RBF: Add a safe mode check period configuration (#6347) Contributed by LiuGuH. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-12-21 12:21:59 +08:00
Jian Zhang
29b0a6c28d
HDFS-17265. RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController (#6298) 2023-12-08 10:18:22 -08:00
LiuGuH
071f850841
HDFS-17261. RBF: Fix getFileInfo return wrong path when get mountTable path which is multi-level (#6288). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-12-01 17:05:58 +05:30
LiuGuH
0417c1c633
HDFS-17263. RBF: Fix client ls trash path cannot get except default nameservices trash path (#6291) Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-11-30 22:46:50 +08:00
Jian Zhang
513e6dcf14
HDFS-17232. RBF: Fix NoNamenodesAvailableException for a long time, when use observer. (#6208) 2023-11-16 09:13:48 -08:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (#6198) 
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
 2023-11-08 14:25:21 +00:00
huhaiyang
4ef2322b6d
HDFS-17243. Add the parameter storage type for getBlocks method (#6238). Contributed by Haiyang Hu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2023-11-06 11:20:25 +08:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
xiaojunxiang
0cfffb3012
HDFS-17214. RBF: The Quota class' andByStorageType method res has an incorrect initial value. (#6135) 
Co-authored-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
 2023-10-03 08:26:03 -07:00
Jian Zhang
d273c13ab5
HDFS-17198. RBF: fix bug of getRepresentativeQuorum when records have same dateModified (#6096) 2023-09-20 10:04:29 -07:00
Jian Zhang
c40a6bd46a
HDFS-17166. RBF: Throwing NoNamenodesAvailableException for a long time, when failover (#5990) 2023-09-05 17:48:27 -07:00
Steve Loughran
28c533a582 Revert "HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977)" 
This reverts commit 1046f9cf9888155c27923f3f56efa107d908ad5b.
 2023-08-31 14:54:53 +01:00
Anmol Asrani
1046f9cf98
HADOOP-18860. Upgrade mockito version to 4.11.0 (#5977) 
As well as the POM update, this patch moves to the (renamed) verify methods. 
Backporting mockito test changes may now require cherrypicking this patch, otherwise
use the old method names.

Contributed by Anmol Asrani
 2023-08-29 12:12:27 +01:00
hchaverri
ad2f45c64f
HDFS-17148. RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL (#5936) 2023-08-11 13:04:32 -07:00
hchaverri
bc48e5cbe8
HDFS-17128. Updating SQLDelegationTokenSecretManager to use LoadingCache so tokens are updated frequently. (#5897) Contributed by Hector Sandoval Chaverri. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-08-08 07:45:14 +08:00
zhangshuyan
5aee0e0c0a
HDFS-17134. RBF: Fix duplicate results of getListing through Router. (#5900). Contributed by Shuyan Zhang. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-01 17:52:54 +08:00
Simbarashe Dzinamarira
ad001c93cf
HDFS-17111. RBF: Optimize msync to only call nameservices that have observer reads enabled. (#5860). Contributed by Simbarashe Dzinamarira. 2023-07-26 13:22:13 +05:30
Viraj Jasani
f0c4286e3e
HDFS-17041. RBF: Fix putAll impl for mysql and file based state stores (#5723) 2023-06-13 09:55:04 -07:00
NishthaShah
3fbadc5d50
HDFS-16946. Fix getTopTokenRealOwners to return String (#5696). Contributed by Nishtha Shah. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-13 10:01:35 +05:30
hfutatzhanghb
35158db711
HDFS-17023. RBF: Record proxy time when call invokeConcurrent method. (#5683). Contributed by farmmamba. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-10 00:06:28 +05:30
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c9676da0896ca864074dfb8fbdefd6e.
 2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
Simbarashe Dzinamarira
d92a5815f4
HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) 2023-05-31 10:20:19 -07:00
hchaverri
124313d215
HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency. (#5691). Contributed by hchaverri. 
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-31 14:26:31 +08:00
Viraj Jasani
3b65b5d68f
HDFS-17020. RBF: mount table addAll should print failed records in std error (#5674) 2023-05-22 18:39:52 -07:00
LiuGuH
f6770dee47
HDFS-16979. RBF: Add proxyuser port in hdfsauditlog (#5552). Contributed by liuguanghua. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 05:02:16 +05:30
Viraj Jasani
8e17385141
HDFS-17009. RBF: state store putAll should also return failed records (#5664) 2023-05-17 09:33:34 -07:00
Viraj Jasani
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points (#5554). Contributed by Viraj Jasani. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-11 08:45:34 +05:30
ZanderXu
4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. (#5200). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-06 08:50:21 +05:30
Viraj Jasani
ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity (#5615) 2023-05-04 09:45:40 -07:00
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
Neil
1ff7a65b9f
HDFS-16954. RBF: The operation of renaming a multi-subcluster directory to a single-cluster directory should throw ioexception. (#5483). Contributed by Max Xie. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-20 15:19:18 +05:30
Simbarashe Dzinamarira
47c22e388e
HDFS-16943. RBF: Implements MySQL based StateStoreDriver. (#5469) 2023-04-05 16:44:29 -07:00
Viraj Jasani
422bf3b24c
HDFS-16973. RBF: MountTableResolver cache size lookup should take read lock (#5533) 2023-04-05 14:06:38 -07:00
Viraj Jasani
937caf7de9
HDFS-16967. RBF: File based state stores should allow concurrent access to the records (#5523) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-04-04 22:39:53 +09:00
Viraj Jasani
b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
Viraj Jasani
f8d0949f7d
HDFS-16953. RBF: Mount table store APIs should update cache only if state store record is successfully updated (#5482) 2023-03-18 14:43:25 -07:00
Viraj Jasani
28d2753d2f
HADOOP-18645. Provide keytab file key name with ServiceStateException (#5433) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-03-01 09:34:12 +08:00
Simbarashe Dzinamarira
61f369c43e
HDFS-16890: RBF: Ensures router periodically refreshes its record of a namespace's state. (#5298) 2023-02-27 17:56:24 +00:00
Owen O'Malley
8025a60ae7
HDFS-16901: Minor fix for unit test. 
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 16:37:49 -08:00
Simbarashe Dzinamarira
4cc33e5e37
HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-22 21:58:44 +00:00
hchaverr
fb31393b65
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 10:38:50 -08:00