big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,244 Commits 2 Branches 2 Tags 264 MiB
aac87ffe76
Commit Graph

271 Commits

Author SHA1 Message Date
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients. 
Fixes #4311

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-09 15:43:28 -07:00
ZanderXu
4a01fadb94
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-09 12:47:36 +05:30
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-05 17:46:15 +05:30
ZanderXu
5567154f71
HDFS-16734. RBF: fix some bugs when handling getContentSummary RPC (#4763) 2022-08-26 16:04:33 -07:00
ZanderXu
8d4f51c432
HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices (#4734). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-24 20:27:15 +05:30
ZanderXu
c37f01d95b
HDFS-16724. RBF should support get the information about ancestor mount points (#4719) 2022-08-23 13:25:42 -07:00
Simba Dzinamarira
a3b1bafa34
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:12:50 -07:00
jianghuazhu
7f176d080c
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:29:31 +09:00
xuzq
e0c8c6eed4
HDFS-16678. RBF should supports disable getNodeUsage() in RBFMetrics (#4606) 2022-08-12 12:01:58 -07:00
xuzq
09cabaad68
HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets (#4531) 2022-08-11 13:23:32 -07:00
xuzq
f80fab2b90
HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout (#4597) 2022-07-28 15:42:53 +08:00
xuzq
2c96357051
HDFS-15079. RBF: Namenode needs to use the actual client Id and callId when going through RBF proxy. (#4530) 2022-07-23 22:19:37 +08:00
slfan1989
2f6916a313
HDFS-16605. Improve Code With Lambda in hadoop-hdfs-rbf moudle. (#4375) 2022-07-21 18:42:55 -07:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
章锡平
f8c7e67fdc
HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. 2022-06-15 21:16:24 +08:00
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Hui Fei
dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen
680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
Felix Nguyen
b4ff49a394
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168) 2022-04-27 14:42:30 +08:00
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
litao
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-03-21 12:40:51 +05:30
Owen O'Malley
7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
YulongZ
48bef285a8
HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971) 2022-02-15 08:44:17 -08:00
YulongZ
3e7a7c3c4a
HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878) 2022-02-08 08:40:57 -08:00
Felix Nguyen
b88d5ae1c6
HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871) 2022-01-10 18:28:20 +08:00
Ayush Saxena
61d424f385
HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-01-04 07:50:01 +05:30
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Ayush Saxena
cab7086fbc
HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745). Contributed by Ayush Saxena. 
Reviewed-by: litao <tomleescut@gmail.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-12-04 10:54:29 +05:30
litao
196935a8d2
HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731). Contributed by tomscut. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-02 19:10:35 +05:30
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Symious
99cb2b6b7f
HDFS-16065. RBF: Add metrics to record Router's operations (#3100) 2021-09-09 18:45:23 +08:00
Symious
c0890e6d04
HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 2021-09-09 09:57:27 +08:00
Eric Yin
de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00