big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,327 Commits 2 Branches 2 Tags 264 MiB
2fbbfe3cc9
Commit Graph

536 Commits

Author SHA1 Message Date
Simbarashe Dzinamarira
f26d8bc9bd
HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 19:46:51 +00:00
Simbarashe Dzinamarira
b56d483258
HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. (#5282) 2023-01-09 23:55:13 +00:00
Simbarashe Dzinamarira
cd19da1309
HDFS-16877: Enables state context for namenode in TestObserverWithRouter (#5257) 2023-01-07 00:18:35 +00:00
Neil
d25c1be517
HDFS-16861. RBF. Truncate API always fails when dirs use AllResolver oder on Router (#5184) 
Co-authored-by: xiezhineng <xiezhineng@corp.netease.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-26 21:19:33 +08:00
ZanderXu
df093ef9af
HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time (#5098) 2022-12-23 14:16:42 +08:00
ZanderXu
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces (#5123) 2022-12-05 16:15:47 +08:00
Owen O'Malley
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 22:12:35 +00:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. (#5142) 2022-11-29 00:49:10 +00:00
Simbarashe Dzinamarira
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-29 00:47:01 +00:00
ZanderXu
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request (#5086) 2022-11-25 09:23:33 +08:00
Owen O'Malley
c71a68ca80
HDFS-16844: Adds resilancy when StateStore gets exceptions. (#5138) 
Allows the StateStore to stay up when there are errors reading the data.
 2022-11-18 17:24:08 +00:00
Simbarashe Dzinamarira
552ee44eba
HDFS-16834: Removes request stateID consistency constraint between clients in different connection pools. (#5121) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
 2022-11-11 15:26:31 +08:00
Simbarashe Dzinamarira
44b8bb7224
HDFS-16821: Fixes regression in HDFS-13522 that enables observer reads by default (#5078) 2022-11-07 08:43:04 -08:00
jianghuazhu
c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. (#5031) 2022-10-20 10:58:23 +08:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
Renukaprasad C
8ce71c8882
[Documentation] RBF: Duplicate statement to be removed for better readabilty (#4881) 2022-09-13 09:24:05 -07:00
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients. 
Fixes #4311

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-09 15:43:28 -07:00
ZanderXu
4a01fadb94
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-09 12:47:36 +05:30
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-05 17:46:15 +05:30
ZanderXu
5567154f71
HDFS-16734. RBF: fix some bugs when handling getContentSummary RPC (#4763) 2022-08-26 16:04:33 -07:00
ZanderXu
8d4f51c432
HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices (#4734). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-24 20:27:15 +05:30
ZanderXu
c37f01d95b
HDFS-16724. RBF should support get the information about ancestor mount points (#4719) 2022-08-23 13:25:42 -07:00
Simba Dzinamarira
a3b1bafa34
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:12:50 -07:00
jianghuazhu
7f176d080c
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:29:31 +09:00
xuzq
622ca0d51f
HDFS-16705. RBF supports healthMonitor timeout configurable and caching NN and client proxy in NamenodeHeartbeatService (#4662) 2022-08-15 13:55:16 -07:00
xuzq
e0c8c6eed4
HDFS-16678. RBF should supports disable getNodeUsage() in RBFMetrics (#4606) 2022-08-12 12:01:58 -07:00
xuzq
521e65acfe
HDFS-16723. Replace incorrect SafeModeException with StandbyException in RouterRpcServer.class (#4716). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-13 00:21:11 +05:30
xuzq
09cabaad68
HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets (#4531) 2022-08-11 13:23:32 -07:00
Steve Vaughan
0fc7dd8228
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) 2022-08-03 09:14:11 -07:00
xuzq
f80fab2b90
HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout (#4597) 2022-07-28 15:42:53 +08:00
xuzq
2c96357051
HDFS-15079. RBF: Namenode needs to use the actual client Id and callId when going through RBF proxy. (#4530) 2022-07-23 22:19:37 +08:00
slfan1989
2f6916a313
HDFS-16605. Improve Code With Lambda in hadoop-hdfs-rbf moudle. (#4375) 2022-07-21 18:42:55 -07:00
wangzhaohui
08a940d5dd
HDFS-16640. RBF: Show datanode IP list when click DN histogram in Router (#4488) 2022-07-21 16:21:31 -07:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
章锡平
f8c7e67fdc
HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. 2022-06-15 21:16:24 +08:00
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
Hui Fei
dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen
680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
Felix Nguyen
b4ff49a394
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168) 2022-04-27 14:42:30 +08:00
Ashutosh Gupta
89279f0b90
HDFS-16255. RBF: Fix dead link to fedbalance document (#4193) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:11:41 +09:00
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
Owen O'Malley
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 09:27:35 -07:00
litao
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-03-21 12:40:51 +05:30
Owen O'Malley
7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
Viraj Jasani
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Neil
2f448acf39
HDFS-16459. RBF: register RBFMetrics in MetricsSystem for promethuessink (#3999) 2022-02-21 11:32:34 -08:00
YulongZ
48bef285a8
HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971) 2022-02-15 08:44:17 -08:00
Neil
c777142ed3
RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. (#3965) 2022-02-10 09:17:30 -08:00
Neil
390967f1f0
RBF: Add search box in Router's tab-mounttable web page (#3968) 2022-02-09 09:17:23 -08:00
YulongZ
3e7a7c3c4a
HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878) 2022-02-08 08:40:57 -08:00
luoyuan3471
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Felix Nguyen
b88d5ae1c6
HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871) 2022-01-10 18:28:20 +08:00
Ayush Saxena
61d424f385
HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-01-04 07:50:01 +05:30
wangzhaohui
8d251bd629
HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 2021-12-21 11:28:57 +00:00
Viraj Jasani
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
Ayush Saxena
cab7086fbc
HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745). Contributed by Ayush Saxena. 
Reviewed-by: litao <tomleescut@gmail.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-12-04 10:54:29 +05:30
litao
196935a8d2
HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731). Contributed by tomscut. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-02 19:10:35 +05:30
litao
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
Symious
89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Viraj Jasani
e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
Symious
d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu
e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Symious
6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Masatake Iwasaki
634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
Ahmed Hussein
62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
Viraj Jasani
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
Xiangyi Zhu
d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 13:53:41 +09:00
Ayush Saxena
263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
Viraj Jasani
3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 15:32:49 +09:00
huhaiyang
68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511) 2021-10-04 23:52:26 +08:00
Viraj Jasani
c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
Viraj Jasani
7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
lfengnan
4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-16 14:58:02 +09:00
litao
2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
litao
4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
Symious
7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 2021-09-12 19:52:47 +08:00
LeonGao
90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Symious
99cb2b6b7f
HDFS-16065. RBF: Add metrics to record Router's operations (#3100) 2021-09-09 18:45:23 +08:00
Symious
c0890e6d04
HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 2021-09-09 09:57:27 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354). Contributed by tomscut. 2021-09-05 14:19:01 +08:00
Viraj Jasani
cc6b545365
HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-04 16:20:08 +05:30
Eric Yin
de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
Viraj Jasani
d9eccb1eb3
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:19:05 +05:30
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
litao
c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Jinglun
e90c41af34 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 19:28:40 +08:00
Jinglun
2b11e160e2 Revert "HDFS-15973." because the user.name and user.email message is not correct. 
This reverts commit b7f69467c1.
 2021-05-30 19:27:10 +08:00
李经纶
b7f69467c1 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 18:50:03 +08:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
zhuxiangyi
e18d806212
HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009) 2021-05-26 13:41:49 +08:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
lfengnan
43bf009112
HDFS-15757 RBF: Improving Router Connection Management (#2651) 2021-05-19 10:53:42 -07:00
Jinglun
1b6994287a HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819). Contributed by zhuobin zheng. 
Reviewed-by: Jinglun <jinglun@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-08 19:50:17 +08:00
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Akira Ajisaka
7c561e2a02
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970) 2021-05-04 00:25:29 +09:00
lfengnan
6e525ab81c
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
lfengnan
552e9dcc6c
HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954). Contributed by Fengnan Li. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
zhuxiangyi
dff95c5eca
HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 10:19:11 +09:00
lfengnan
17be99f9f9
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903) 2021-04-15 16:11:47 -07:00
Ayush Saxena
7d6f0ca0bd Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." 
This reverts commit e565b05c80.

The removed method needs to be used in HDFS-15423.
 2021-04-13 23:28:42 +05:30
Inigo Goiri
82462739f8
Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) 
This reverts commit cb3ed32fe0.
 2021-04-12 20:44:31 -07:00
lfengnan
cb3ed32fe0
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605) 2021-04-12 12:42:33 -07:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
He Xiaoqiao
da4ceba4aa
HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C. 2021-03-31 13:00:13 +08:00
Harunobu Daikoku
ea6595d3b6
HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
 2021-03-29 11:43:29 +09:00
litao
e565b05c80
HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut. 2021-03-14 04:07:31 +05:30
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
Ayush Saxena
2ce5752fa8 HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun. 2021-02-20 19:34:29 +05:30
Akira Ajisaka
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Akira Ajisaka
2b3c5b1733
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-02-16 14:50:50 +09:00
Zehao Chen
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
Ayush Saxena
e40f99f6d5 HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun. 2021-01-26 14:25:27 +05:30
Yiqun Lin
0d7ac54510 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 2021-01-12 23:38:55 +08:00
Ayush Saxena
2ba7ec2b48 HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun. 2021-01-08 16:27:54 +05:30
Ayush Saxena
77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
Ayush Saxena
5ab1db9e21 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 2020-12-21 09:58:30 +05:30
Ayush Saxena
2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ayush Saxena
7dda804a1a HDFS-14090. RBF: Improved isolation for downstream name nodes. {Static}. Contributed by Fengnan Li and CR Hota. 2020-12-04 22:56:37 +05:30
Masatake Iwasaki
6a1d7d9ed2
HDFS-15677. TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk. (#2503) 2020-12-01 08:06:52 +09:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
Hui Fei
f1d5652359 HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang. 2020-11-10 13:09:03 +08:00
Akira Ajisaka
b3ba74d72d
HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 2020-10-28 19:12:23 +09:00
Hui Fei
264c948e60 HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang 2020-10-23 14:06:22 +08:00
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Akira Ajisaka
69ef9b1ee8
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 09:52:12 +09:00
Hui Fei
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 2020-10-09 16:12:22 +08:00
Akira Ajisaka
074f0d46af
HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360) 
* Support getHAServiceState in DFSRouter
 2020-10-07 13:37:45 +09:00
hemanthboyina
921ca1f554 HDFS-15543. RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled. Contributed by Hemanth Boyina. 2020-10-07 09:58:53 +05:30
huangtianhua
a89ca56a1b
HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347) 2020-09-29 19:00:55 -07:00
Ayush Saxena
d686a655bc HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui. 2020-09-29 00:03:11 +05:30
Ayush Saxena
fc8a6dd8a9 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 2020-09-28 23:55:31 +05:30
Inigo Goiri
eae0035a2d HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao. 2020-09-28 09:40:39 -07:00
Aryan Gupta
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-09-24 21:15:55 +09:00
Hui Fei
474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
lfengnan
3e8b1e7426
HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266). Contributed by Fengnan Li. 2020-09-21 23:24:13 +05:30