big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,640 Commits 2 Branches 2 Tags 264 MiB
ff8eac517a
Commit Graph

9872 Commits

Author SHA1 Message Date
jianghuazhu
78cc528739
HDFS-17012.Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT. (#5659). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 14:27:19 +05:30
Tsz-Wo Nicholas Sze
350dcaf616
HDFS-17010. Add a subtree test to TestSnapshotDiffReport. (#5656) 2023-05-18 15:53:26 +08:00
wangzhaohui
03163f9de2
HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 12:12:11 +05:30
Viraj Jasani
8e17385141
HDFS-17009. RBF: state store putAll should also return failed records (#5664) 2023-05-17 09:33:34 -07:00
liang3zy22
482897a0f6
Fix typos in HDFS documents. (#5665) 2023-05-17 09:28:01 -07:00
smarthan
251439d769
HDFS-16985. Fix data missing issue when delete local block file. (#5564). Contributed by Chengwei Wang. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:33:38 +08:00
Viraj Jasani
5d0cc455f5
HDFS-17008. Fix RBF JDK 11 javadoc warnings (#5648) 2023-05-12 17:27:13 -07:00
zhtttylz
0c77629849
HDFS-17001. Support getStatus API in WebHDFS (#5628). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 03:49:13 +05:30
Viraj Jasani
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points (#5554). Contributed by Viraj Jasani. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-11 08:45:34 +05:30
zhtttylz
5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API (#5602). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 23:10:19 +05:30
slfan1989
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
zhangshuyan
03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). (#5622). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-09 10:05:28 +08:00
ZanderXu
4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. (#5200). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-06 08:50:21 +05:30
Viraj Jasani
ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity (#5615) 2023-05-04 09:45:40 -07:00
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
SevenAddSix
1079890ae3
HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus (#4665). Contributed by Jiale Qi. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-01 21:22:26 +05:30
fanluoo
1a2cd965a7
HDFS-16897. Fix abundant Broken pipe exception in BlockSender (#5329). Contributed by fanluo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:24:56 +05:30
wangzhaohui
0e63152218
HDFS-16995. Remove unused parameters at NameNodeHttpServer#initWebHdfs (#5601). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-28 05:25:01 +05:30
Tsz-Wo Nicholas Sze
d9576bb9ee
HDFS-16972. Delete a snapshot may deleteCurrentFile. (#5532) 2023-04-27 09:17:47 -07:00
cxzl25
5af0845076
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-26 02:11:11 +05:30
Sebastian Baunsgaard
6aac6cb212
HADOOP-18660. Filesystem Spelling Mistake (#5475). Contributed by Sebastian Baunsgaard. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 21:44:04 +05:30
zhtttylz
c9e0af9961
HDFS-16981. Support getFileLinkStatus API in WebHDFS (#5572). Contributed by Hualong Zhang. 
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 03:30:56 +05:30
Tsz-Wo Nicholas Sze
dc78849f27
HDFS-16975. FileWithSnapshotFeature.isCurrentFileDeleted is not reloaded from FSImage. (#5546) 2023-04-24 09:04:28 -07:00
zhangshuyan
6a23c376c9
HDFS-16986. EC: Fix locationBudget in getListing(). (#5582). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-24 18:53:25 +08:00
wangzhaohui
51dcbd1d61
HDFS-16988. Improve NameServices info at JournalNode web UI (#5584). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:13:02 +05:30
Neil
1ff7a65b9f
HDFS-16954. RBF: The operation of renaming a multi-subcluster directory to a single-cluster directory should throw ioexception. (#5483). Contributed by Max Xie. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-20 15:19:18 +05:30
Steve Loughran
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
zhangshuyan
0185afafea
HDFS-16974. Consider volumes average load of each DataNode when choosing target. (#5541). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-14 10:33:30 +08:00
Melissa You
2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 (#5536) 2023-04-13 09:07:42 -07:00
rdingankar
3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… (#5495) 2023-04-10 08:56:00 -07:00
zhtttylz
523ff81624
HDFS-16952. Support getLinkTarget API in WebHDFS (#5517) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-06 19:44:47 +08:00
Simbarashe Dzinamarira
47c22e388e
HDFS-16943. RBF: Implements MySQL based StateStoreDriver. (#5469) 2023-04-05 16:44:29 -07:00
Viraj Jasani
422bf3b24c
HDFS-16973. RBF: MountTableResolver cache size lookup should take read lock (#5533) 2023-04-05 14:06:38 -07:00
Viraj Jasani
937caf7de9
HDFS-16967. RBF: File based state stores should allow concurrent access to the records (#5523) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-04-04 22:39:53 +09:00
zhtttylz
811441d5bc
HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc (#5491) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-01 18:18:20 +08:00
Viraj Jasani
b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
zhangshuyan
700147b4ac
HDFS-16964. Improve processing of excess redundancy after failover. (#5510). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-28 16:14:59 +08:00
Kidd5368
5cf62d1498
HDFS-16948. Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. (#5474). Contributed by Kidd53685368. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 17:00:23 +08:00
Ayush Saxena
e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Viraj Jasani
0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem (#5492) 2023-03-21 10:23:16 +08:00
Viraj Jasani
f8d0949f7d
HDFS-16953. RBF: Mount table store APIs should update cache only if state store record is successfully updated (#5482) 2023-03-18 14:43:25 -07:00
Viraj Jasani
b6a9d7b442
HADOOP-18631. (ADDENDUM) Use LogCapturer to match audit log pattern and remove hdfs async audit log configs (#5451) 2023-03-18 06:33:50 +08:00
Viraj Jasani
15935fa865
HDFS-16947. RBF NamenodeHeartbeatService to report error for not being able to register namenode in state store (#5470) 2023-03-16 00:59:55 +08:00
Stephen O'Donnell
eee2ea075d
HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/ 2023-03-15 10:03:00 +05:30
Stephen O'Donnell
ca6f5afb6d
HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460) 2023-03-11 16:40:07 +00:00
slfan1989
927401886a
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434) 
Contributed by Shilun Fan
 2023-03-06 15:26:53 +00:00
zhangshuyan
2cb0c35fc1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-06 20:10:31 +08:00
ZanderXu
6bd2444815
HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-03-01 16:18:38 -08:00
Tom
162288bc0a
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) 
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations. 
ignoredNodes list is only used on hedged read codepath

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-01 19:47:04 +00:00
Viraj Jasani
2ab7eb4caa
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:10 +00:00