big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,087 Commits 2 Branches 2 Tags 264 MiB
02aedd7811
Commit Graph

271 Commits

Author SHA1 Message Date
Melissa You
02aedd7811
Hadoop-18519. Backport HDFS-15383 and HADOOP-17835 to branch-3.3 (#5112) 
* HDFS-15383. RBF: Add support for router delegation token without watch (#2047)

Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.

* HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: lfengnan <lfengnan@uber.com>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
 2022-11-07 13:29:50 -08:00
Xing Lin
760144f135
HDFS-16628. RBF: Correct target directory when move to trash for kerberos login user. (#4974) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-11 16:14:39 +09:00
Xing Lin
7d7f7a9e9b
HDFS-16024. RBF: Rename data to the Trash should be based on src location (#4962) 
(cherry picked from commit e18d806212)

Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:33:48 +09:00
Simba Dzinamarira
e28dc524f6
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:27:21 -07:00
Steve Vaughan
a73efb2d55
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) (#4790) 2022-08-22 20:26:03 -07:00
jianghuazhu
2123859d60
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f176d080c)
 2022-08-21 07:31:01 +09:00
Ayush Saxena
df4e59318f
HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun. 
Only Prod Changes: Test already cherry-picked as part of HDFS-16310 via (496657c63f)
 2022-07-23 23:48:02 +05:30
Viraj Jasani
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes (#4107) (#4259) 2022-05-06 04:55:46 +08:00
Owen O'Malley
1175acbb75 HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 10:24:54 -07:00
Owen O'Malley
c52b97d084 HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:30:29 -07:00
litao
496657c63f HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 
Cherry-picked from 5b05068f by Owen O'Malley
 2022-03-14 10:29:30 -07:00
Hui Fei
2479d4ab6c HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang 
Cherry-picked from 264c948e by Owen O'Malley
 2022-03-14 10:29:04 -07:00
Hui Fei
8e129e5b8d HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 
Cherry-picked from 518a212c by Owen O'Malley
 2022-03-14 10:28:55 -07:00
luoyuan3471
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
wangzhaohui
cc7b7e174c HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 11:46:23 +00:00
Ayush Saxena
5b1cf17f88
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:53 +05:30
Symious
c0f32f3cf8 HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 
(cherry picked from commit 7f6553af75)
 2021-09-13 10:48:09 +08:00
Symious
8affaa6312 HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 
(cherry picked from commit c0890e6d04)
 2021-09-10 10:01:27 +08:00
Viraj Jasani
ef3a2a3b3b HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:52 +05:30
litao
02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a)
 2021-06-08 16:59:59 +09:00
Wei-Chiu Chuang
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Wei-Chiu Chuang
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
Aryan Gupta
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
 2021-05-13 12:16:17 +08:00
lfengnan
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c)
 2021-05-02 19:19:55 +09:00
Wei-Chiu Chuang
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029)
 2021-05-02 11:12:37 +09:00
litao
0e9042fed5 HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit b968fa0957)
 2021-04-28 18:52:05 +09:00
Ayush Saxena
f809b863fe HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 
(cherry picked from commit 6e416a83d1)
 2021-04-28 18:51:43 +09:00
Ayush Saxena
3c7433f2a1 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 
(cherry picked from commit 8d49229c37)
 2021-04-28 18:51:37 +09:00
zhuxiangyi
2752156e3e HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit dff95c5eca)
 2021-04-23 11:22:52 +09:00
Ayush Saxena
2da3356428 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 
(cherry picked from commit fc8a6dd8a9)
 2021-04-02 17:13:00 +08:00
Takanobu Asanuma
f5cc1540b4 HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina. 
(cherry picked from commit ca8e7a7725)
 2021-04-02 17:11:33 +08:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c37)
 2021-04-02 17:11:16 +08:00
Inigo Goiri
327488d688 HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 
(cherry picked from commit 80b877a72f)
 2021-04-02 14:12:17 +08:00
Akira Ajisaka
53b024a2b7 HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module 
(cherry picked from commit 9062814bab)
 2021-03-30 22:29:49 +08:00
Harunobu Daikoku
db8af483d1 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b6)
 2021-03-29 11:47:16 +09:00
Ayush Saxena
e4217be749 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 
(cherry picked from commit 5ab1db9e21)
 2021-03-24 17:44:29 +08:00
Zehao Chen
1768c0096f HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 17:32:14 +00:00
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Akira Ajisaka
89314a7bae
HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360) 
* Support getHAServiceState in DFSRouter

(cherry picked from commit 074f0d46af)
 2020-10-07 13:40:31 +09:00
Hui Fei
ed19f63998
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.

(cherry picked from commit 474fa80bfb)
 2020-09-23 15:39:51 +09:00
Akira Ajisaka
0731c8b5d0
HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267) 
(cherry picked from commit c78d18023d)
 2020-09-14 11:36:26 +09:00
hemanthboyina
ac0a377779 HDFS-15536. RBF: Clear Quota in Router was not consistent. 
(cherry picked from commit 6e618b6a7e)
 2020-08-26 13:04:47 +05:30
hemanthboyina
b5d712251c HDFS-15503. File and directory permissions are not able to be modified from WebUI. 
(cherry picked from commit 82f3ffcd64)
 2020-08-03 23:43:29 +05:30
Vinayakumar B
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Surendra Singh Lilhore
a6c718fd0f HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 8e6227441a)
 2020-04-18 20:21:01 +05:30
Akira Ajisaka
d501c6a43d
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 
(cherry picked from commit 1695d8d59c)
 2020-04-04 00:16:52 +09:00
Inigo Goiri
f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena
cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Inigo Goiri
679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
He Xiaoqiao
5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 22:41:31 -07:00