big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,804 Commits 2 Branches 2 Tags 264 MiB
6e1df1c048
Commit Graph

321 Commits

Author SHA1 Message Date
Ye Ni
e820baa6e6
HDFS-15417. RBF: Get the datanode report from cache for federation WebHDFS operations (#2080) 2020-07-06 16:17:09 -07:00
lfengnan
84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch (#2047) 
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
 2020-06-23 13:12:29 -07:00
Yiqun Lin
9cbd76cc77 HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Yiqun Lin
1983eea62d HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun. 2020-05-20 10:39:40 +08:00
Ayush Saxena
a3809d2023 HDFS-15082. RBF: Check each component length of destination path when add/update mount entry. Contributed by Xiaoqiao He. 2020-05-17 19:45:34 +05:30
Ayush Saxena
6e416a83d1 HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 2020-05-17 03:06:27 +05:30
Ayush Saxena
936bf09c37 HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 2020-05-12 21:54:54 +05:30
Xiaoyu Yao
047d8879e7
HDFS-15345. RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. 2020-05-12 08:31:04 -07:00
S O'Donnell
29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Akira Ajisaka
e069a06137
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 2020-04-20 08:41:18 -07:00
Surendra Singh Lilhore
8e6227441a HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 2020-04-18 20:07:21 +05:30
Ayush Saxena
cc5c1da7c1 HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang. 2020-04-15 23:01:58 +05:30
Ayush Saxena
8d49229c37 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-04-12 12:05:50 +05:30
Ayush Saxena
275c478330 HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena. 2020-04-09 23:44:36 +05:30
Akira Ajisaka
1695d8d59c
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 2020-04-04 00:13:41 +09:00
Ayush Saxena
4a3eb10972 HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He. 2020-04-02 11:06:01 +05:30
Inigo Goiri
80b877a72f HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 2020-03-30 12:29:21 -07:00
Brahma Reddy Battula
8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Inigo Goiri
f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena
cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Inigo Goiri
679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
He Xiaoqiao
5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 22:41:31 -07:00
xiaoxiaopan118
439d935e1d
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 2020-02-17 15:55:10 +09:00
Akira Ajisaka
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router (#1832) 
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
 2020-02-13 10:06:07 +09:00
Ayush Saxena
3df0adaaea HDFS-15127. RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Contributed by Inigo Goiri 2020-02-12 19:41:04 +05:30
Vinayakumar B
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Ayush Saxena
92c58901d7 HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena 2020-01-23 18:18:34 +05:30
Inigo Goiri
263413e838 HDFS-15112. RBF: Do not return FileNotFoundException when a subcluster is unavailable. 2020-01-16 10:39:14 -08:00
Chao Sun
d7c4f8ab21 HDFS-13616. Batch listing of multiple directories (#1725) 2020-01-15 17:22:47 -08:00
Ayush Saxena
7b62409ace HDFS-15108. RBF: MembershipNamenodeResolver should invalidate cache incase of active namenode update. Contributed by Ayush Saxena. 2020-01-11 10:21:31 +05:30
Akira Ajisaka
0315ef8448 HDFS-15100. RBF: Print stacktrace when DFSRouter fails to fetch/parse JMX output from NameNode. (#1800) 2020-01-10 13:16:57 +09:00
Ayush Saxena
b32757c616 HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena. 2020-01-10 08:14:56 +05:30
Ayush Saxena
8fe01db34a HDFS-15094. RBF: Reuse ugi string in ConnectionPoolID. Contributed by Ayush Saxena. 2020-01-09 09:02:38 +05:30
Ayush Saxena
fd30f4c52b HDFS-15096. RBF: GetServerDefaults Should be Cached At Router. Contributed by Ayush Saxena. 2020-01-09 08:26:51 +05:30
Takanobu Asanuma
4a76ab777f HDFS-15090. RBF: MountPoint Listing Should Return Flag Values Of Destination. Contributed by Ayush Saxena. 2020-01-06 18:09:59 +09:00
luhuachao
77ae7b9ce2 HDFS-15089. RBF: SmallFix for RBFMetrics in doc (#1786) 2020-01-06 12:31:13 +09:00
Ayush Saxena
8730a7bf60 HDFS-15003. RBF: Make Router support storage type quota. Contributed by Jinglun. 2019-12-27 09:22:58 +05:30
Ayush Saxena
72aee114f8 HDFS-15053. RBF: Add permission check for safemode operation. Contributed by Xiaoqiao He. 2019-12-14 17:57:23 +05:30
Inigo Goiri
7fe924b1c0 HDFS-15016. RBF: getDatanodeReport() should return the latest update. Contributed by Inigo Goiri. 2019-12-13 10:51:14 -08:00
Takanobu Asanuma
93bb368094 HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li 2019-12-12 10:56:19 +09:00
Takanobu Asanuma
c4733377d0
HDFS-15040. RBF: Secured Router should not run when SecretManager is not running. (#1745) 2019-12-10 16:03:32 +09:00
Akira Ajisaka
9f09852051
HDFS-15043. RBF: The detail of the Exception is not shown in ZKDelegationTokenSecretManagerImpl (#1748) 2019-12-10 16:00:20 +09:00
Yiqun Lin
47fdae7904 HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun. 2019-12-04 18:19:11 +08:00
Ayush Saxena
6b2d6d4aaf HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina. 2019-11-29 23:55:37 +05:30
Ayush Saxena
c422e36397 HDFS-13842. RBF: Exceptions are conflicting when creating the same mount entry twice. Contributed by Ranith Sardar. 2019-11-22 11:42:19 +05:30
Ayush Saxena
b89fd4dfe9 HDFS-15002. RBF: Fix annotation in RouterAdmin. Contributed by Jinglun. 2019-11-22 01:08:11 +05:30
Ayush Saxena
98d249dcda HDFS-14996. RBF: GetFileStatus fails for directory with EC policy set in case of multiple destinations. Contributed by Ayush Saxena. 2019-11-21 12:08:42 +05:30
Ayush Saxena
12617fad2e HDFS-14955. RBF: getQuotaUsage() on mount point should return global quota. Contributed by Jinglun. 2019-11-18 16:04:52 +05:30
Ayush Saxena
3b5a0e86c1 HDFS-14974. RBF: Make tests use free ports. Contributed by Inigo Goiri. 2019-11-18 15:49:36 +05:30
Wang Yuxuan
b25a37c322 HDFS-14962. RBF: ConnectionPool#newConnection() error log wrong protocol class (#1699). Contributed by Yuxuan Wang. 2019-11-11 10:48:38 +05:30
Takanobu Asanuma
6663d6a5c2 HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li 2019-11-11 14:04:57 +09:00
Ayush Saxena
42fc8884ab HDFS-14815. RBF: Update the quota in MountTable when calling setQuota on a MountTable src. Contributed by Jinglun. 2019-11-08 08:48:52 +05:30
Takanobu Asanuma
7901062707 HDFS-14891. RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme. Contributed by Xieming Li 2019-11-05 15:03:09 +09:00
Inigo Goiri
f18bbdd9d8 HDFS-14927. RBF: Add metrics for async callers thread pool. Contributed by Leon Gao. 2019-11-01 10:14:31 -07:00
Ayush Saxena
ff6a492d66 HDFS-14916. RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'. Contributed by Xieming Li. 2019-10-21 18:20:00 +05:30
Inigo Goiri
e6f95eb0f7 HDFS-14887. RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable. Contributed by hemanthboyina. 2019-10-19 13:24:58 -07:00
Ayush Saxena
375224edeb HDFS-14739. RBF: LS command for mount point shows wrong owner and permission information. Contributed by Jinglun. 2019-10-16 19:26:01 +05:30
Ayush Saxena
6e5cd5273f HDFS-14899. Use Relative URLS in Hadoop HDFS RBF. Contributed by David Mollitor. 2019-10-12 13:26:47 +05:30
Ayush Saxena
761594549e HDFS-14814. RBF: RouterQuotaUpdateService supports inherited rule. Contributed by Jinglun. 2019-10-08 14:01:44 +05:30
Ayush Saxena
5a7483ca5c HDFS-14888. RBF: Enable Parallel Test Profile for builds. Contributed by Ayush Saxena. 2019-10-03 19:21:27 +05:30
Inigo Goiri
b1e55cfb55 HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He. 2019-09-26 15:26:23 -07:00
Takanobu Asanuma
bdaaa3bbf2 HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina 2019-09-25 23:48:49 +09:00
Wei-Chiu Chuang
43203b466d HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-24 09:38:58 -07:00
Vinayakumar B
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes (#1494). Contributed by Vinayakumar B. 2019-09-23 12:37:47 +05:30
Ayush Saxena
aa938662f9 HDFS-14833. RBF: Router Update Doesn't Sync Quota. Contributed by Ayush Saxena. 2019-09-21 09:53:13 +05:30
Vinayakumar B
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432) 
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
 2019-09-20 16:08:30 +05:30
Inigo Goiri
a79f286c6f HDFS-14609. RBF: Security should use common AuthenticationFilter. Contributed by Chen Zhang. 2019-09-19 11:00:10 -07:00
Takanobu Asanuma
c255333e20 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 2019-09-11 16:54:08 +09:00
Takanobu Asanuma
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. (#1414) 2019-09-11 10:32:07 +09:00
Ayush Saxena
2f70b52a5b HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq. 2019-09-05 08:29:56 +05:30
Surendra Singh Lilhore
05704754a0 HDFS-14777. RBF: Set ReadOnly is failing for mount Table but actually readonly succeed to set. Contributed by Ranith Sardar. 2019-09-04 15:09:09 +05:30
Ayush Saxena
040f6e93bb HDFS-14654. RBF: TestRouterRpc#testNamenodeMetrics is flaky. Contributed by Chen Zhang. 2019-09-02 13:01:52 +05:30
Ayush Saxena
c3abfcefdd HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena. 2019-09-02 07:24:04 +05:30
Ayush Saxena
18d74fe41c HDFS-14711. RBF: RBFMetrics throws NullPointerException if stateStore disabled. Contributed by Chen Zhang. 2019-09-01 12:53:10 +05:30
Ayush Saxena
8e779a151e HDFS-14721. RBF: ProxyOpComplete is not accurate in FederationRPCPerformanceMonitor. Contributed by xuzq. 2019-08-29 20:08:38 +05:30
Surendra Singh Lilhore
29bd6f3fc3 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. 2019-08-28 23:58:23 +05:30
Inigo Goiri
48cb583906 HDFS-14710. RBF: Improve some RPC performance by using previous block. Contributed by xuzq. 2019-08-28 10:48:00 -07:00
Takanobu Asanuma
b69ac575a1 HDFS-14772. RBF: hdfs-rbf-site.xml can't be loaded automatically. Contributed by Yuxuan Wang 2019-08-27 14:32:32 +09:00
Ayush Saxena
d2225c8ca8 HDFS-14722. RBF: GetMountPointStatus should return mountTable information when getFileInfoAll throw IOException. Contributed by xuzq. 2019-08-24 08:47:43 +05:30
Inigo Goiri
0b796754b9 HDFS-14766. RBF: MountTableStoreImpl#getMountTableEntries returns extra entry. Contributed by Chen Zhang. 2019-08-23 14:53:57 -07:00
Wang Yuxuan
894e2300d6 HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly (#1334) 
HDFS-14761. RBF: MountTableResolver cannot invalidate cache correctly
 2019-08-23 10:46:39 -07:00
Ayush Saxena
c92de8209d HDFS-14747. RBF: IsFileClosed should be return false when the file is open in multiple destination. Contributed by xuzq. 2019-08-23 19:08:07 +05:30
Ayush Saxena
52c77bc160 HDFS-14741. RBF: RecoverLease should be return false when the file is open in multiple destination. Contributed by xuzq 2019-08-22 08:57:22 +05:30
Ayush Saxena
f9029c4070 HDFS-14744. RBF: Non secured routers should not log in error mode when UGI is default. Contributed by CR Hota. 2019-08-22 02:40:39 +05:30
Inigo Goiri
93595febaa HDFS-14756. RBF: getQuotaUsage may ignore some folders. Contributed by Chen Zhang. 2019-08-21 09:39:57 -07:00
Inigo Goiri
5eeb6da2d4 HDFS-14714. RBF: implement getReplicatedBlockStats interface. Contributed by Chen Zhang. 2019-08-21 09:38:17 -07:00
Sunil G
bd92462321 HDFS-14729. Upgrade Bootstrap and jQuery versions used in HDFS UIs. Contributed by Vivek Ratnavel Subramanian. This closes #1297 2019-08-20 18:42:19 +05:30
Ayush Saxena
b06c2345ef HDFS-14713. RBF: RouterAdmin supports refreshRouterArgs command but not on display. Contributed by wangzhaohui. 2019-08-14 20:44:31 +05:30
Ayush Saxena
89b102f916 HDFS-14661. RBF: updateMountTableEntry shouldn't update mountTableEntry if targetPath not exist. Contributed by xuzq. 2019-08-01 08:43:39 +05:30
Chao Sun
b008072044 HDFS-14681. RBF: TestDisableRouterQuota failed because port 8888 was occupied. 2019-07-31 10:24:48 -07:00
Inigo Goiri
7849bdcf70 HDFS-14449. Expose total number of DT in JMX for Namenode. Contributed by Fengnan Li. 2019-07-30 13:45:27 -07:00
CR Hota
611718f59f HDFS-14670: RBF: Create secret manager instance using FederationUtil#newInstance. 2019-07-29 11:00:22 -07:00
Ayush Saxena
6282c02d7d HDFS-14577. RBF: FederationUtil#newInstance should allow constructor without context. Contributed by CR Hota. 2019-07-19 17:44:13 +05:30
Takanobu Asanuma
9792f58662 HDFS-14653. RBF: Correct the default value for dfs.federation.router.namenode.heartbeat.enable. Contributed by Ayush Saxena. 2019-07-18 10:16:20 +09:00
Ayush Saxena
64d4abf489 HDFS-14593. RBF: Implement deletion feature for expired records in State Store. Contributed by Takanobu Asanuma. 2019-07-15 22:38:00 +05:30
Inigo Goiri
c3ca348b81 HDFS-14620. RBF: Fix 'not a super user' error when disabling a namespace in kerberos with superuser principal. Contributed by luhuachao. 2019-07-04 11:21:55 -07:00
Inigo Goiri
041e0c0564 HDFS-14036. RBF: Add hdfs-rbf-default.xml to HdfsConfiguration by default. Contributed by Takanobu Asanuma. 2019-06-27 09:57:52 -07:00
Inigo Goiri
719d57bf46 HDFS-14545. RBF: Router should support GetUserMappingsProtocol. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Akira Ajisaka
8627f6532b HDFS-14550. RBF: Failed to get statistics from NameNodes before 2.9.0. Contributed by He Xiaoqiao. 2019-06-24 22:03:04 +05:30