big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,037 Commits 2 Branches 2 Tags 264 MiB
37d6cada14
Commit Graph

5469 Commits

Author SHA1 Message Date
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
dingshun3016
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. (#5180) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-09 08:10:04 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
huhaiyang
e9319e696c
HDFS-16811. Support DecommissionBackoffMonitor Parameters reconfigurablereconfigurable (#5068) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-04 14:18:59 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:03:15 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-10-27 14:39:01 +09:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (#4750) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-17 18:56:15 +08:00
huhaiyang
d14b88c698
HDFS-16774.Improve async delete replica on datanode (#4903) 
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException

Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-11 16:22:56 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers (#4979) 
Contributed by P J Fanning
 2022-10-10 18:56:35 +01:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917) 
Contributed-by: navinko <nakumr@cloudera.com>
 2022-09-30 23:28:12 +08:00
ZanderXu
e68006cd70
HDFS-16772. refreshHostsReader should use the latest configuration (#4890) 2022-09-19 13:27:07 -07:00
ZanderXu
a73c4804d8
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs (#4871) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-16 10:09:01 -07:00
ZanderXu
43c1ebae16
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException (#4882) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-15 12:44:36 -07:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
caozhiqiang
1923096adb
Allow block reconstruction pending timeout to be refreshable (#4567) 
Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-12 11:45:01 +09:00
ZanderXu
c947c326e8
HDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger than HighestWrittenTxId (#4560) 
Co-authored-by: Zander Xu <zengqiang.xu@shopee.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-09-06 10:12:55 -07:00
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-05 17:46:15 +05:30
ZanderXu
ac42519ade
HDFS-16593. Correct the BlocksRemoved metric on DataNode side (#4353). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-09-05 19:35:48 +08:00
Steve Vaughan
2dd8b1342e
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833) 
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr
 2022-09-01 14:00:15 +01:00
zhengchenyu
231a4468cd
HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-08-25 10:37:25 -07:00
ZanderXu
1691cccc89
HDFS-16738. Invalid CallerContext caused NullPointerException (#4791) 2022-08-25 17:12:27 +08:00
Steve Vaughan
6fbc38db95
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724) 2022-08-23 08:03:57 -07:00
Steve Vaughan
a9e5fb3313
HDFS-16684. Exclude the current JournalNode (#4723) 
Exclude bound local addresses, including the use of a wildcard address in the bound host configurations.
* Allow sync attempts with unresolved addresses
* Update the comments.
* Remove unused import

Signed-off-by: stack <stack@apache.org>
 2022-08-22 09:52:45 -07:00
Hui Fei
86cc96c493
Revert "HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628)" (#4743) 2022-08-15 19:48:05 +08:00
xuzq
59619ad247
HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628) 2022-08-12 12:19:28 +08:00
Steve Vaughan
2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
xuzq
895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
zhangshuyan0
dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
slfan1989
06ac327e88
HDFS-16619. Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Import (#4406) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-07-28 11:07:24 +08:00
xuzq
01a2e0f6bd
HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. (#4155) 
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-07-26 04:30:00 +08:00
Neil
2f49eec5dd
HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output (#4541). Contributed by Max Xie. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-07-25 17:39:25 +08:00
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) 2022-07-16 04:52:12 +08:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Ashutosh Gupta
b7edc6c60c
HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-24 18:35:00 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
Viraj Jasani
7a1d811197
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:42:43 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00
caozhiqiang
9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu
6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:18:58 +08:00
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
jianghuazhu
4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-22 21:59:38 +08:00
Viraj Jasani
93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:28:59 +08:00
singer-bin
d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Viraj Jasani
6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
daimin
d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
daimin
29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
Viraj Jasani
b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack
9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
 2022-04-28 10:28:01 -07:00
stack
4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)" 
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
 2022-04-28 10:26:46 -07:00
huaxiangsun
bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
 2022-04-28 10:24:37 -07:00
smarthan
f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
litao
f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Hui Fei
17d64ba495
Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232) 2022-04-24 17:18:31 +08:00
litao
acc0e0a210
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035) 2022-04-24 14:54:41 +08:00
litao
5ebbacc480
HDFS-16552. Fix NPE for TestBlockManager (#4210) 2022-04-23 12:14:06 +09:00
S O'Donnell
a4683be65e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit dbeeee0363.
 2022-04-20 20:34:43 +01:00
qinyuren
76bbd17374
HDFS-16544. EC decoding failed due to invalid buffer (#4179) 2022-04-20 15:04:27 +09:00
qinyuren
52e152f8b0
HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
 2022-04-19 13:37:28 +09:00
jianghuazhu
900682e712
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 11:21:51 +08:00
Quanlong Huang
35d4c02bcc
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
 2022-04-18 10:32:29 +08:00
Stephen O'Donnell
dbeeee0363
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 2022-04-17 20:05:11 +08:00
He Xiaoqiao
cb975c3df6
HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141) Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-04-17 19:21:27 +08:00
Renukaprasad C
f14f305051
HDFS-16526. Add metrics for slow DataNode (#4162) 2022-04-15 21:37:05 +05:30
caozhiqiang
cee8c62498
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) 2022-04-14 18:42:39 +09:00
daimin
c65c383b7e
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 2022-04-14 11:07:06 +08:00
Takanobu Asanuma
2efab92959
HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 2022-04-14 11:23:38 +09:00
qinyuren
45394433a1
HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 2022-04-13 11:27:37 +09:00
litao
5412fbf6d4
HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001) 2022-04-08 12:26:39 +08:00
singer-bin
807a428b55
HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069) 2022-04-08 10:27:35 +09:00
Viraj Jasani
f70935522b
HADOOP-18188. Support touch command for directory (#4135) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-07 17:29:45 +09:00
Viraj Jasani
7c20602b17
HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 18:17:02 +09:00
wangzhaohui
61bbdfd3a7
HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 17:29:43 +09:00
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
Xing Lin
15a5ea2c95
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
 2022-03-31 21:59:09 +00:00
He Xiaoqiao
2bf78e2416
HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mode. (#4085). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-31 14:00:38 +08:00
litao
ac50657c37
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) 2022-03-31 00:51:17 +09:00
He Xiaoqiao
6eea28c3f3
HDFS-16498. Fix NPE for checkBlockReportLease #4057. Contributed by tomscut. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-30 14:56:04 +08:00
Lei Yang
a9b43966c0
HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:15:28 -07:00
GuoPhilipse
046a6204b4
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-27 22:23:48 +09:00
He Xiaoqiao
ef8bff0df9
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-23 11:32:12 +08:00
He Xiaoqiao
26ba3846cc
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang." 
This reverts commit 88975496d8.
 2022-03-23 11:01:35 +08:00
He Xiaoqiao
88975496d8
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-22 22:28:09 +08:00
Owen O'Malley
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 09:27:35 -07:00
litao
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-03-21 12:40:51 +05:30
He Xiaoqiao
62d59e516e
HDFS-16504. Add parameter for NameNode to process getBloks request. (#4068). Contributed by Max Xie. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-20 14:01:45 +08:00
litao
7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058) 2022-03-17 12:04:48 +08:00
Viraj Jasani
1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-16 09:32:29 +08:00
Xing Lin
8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 11:29:48 -07:00
He Xiaoqiao
a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
 2022-03-12 18:40:09 +08:00
Stephen O'Donnell
49e4e4d0a5
HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error (#4051) 2022-03-08 11:07:11 +00:00
Viraj Jasani
278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 2022-03-04 22:17:48 +05:30
litao
9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041) 2022-03-02 11:25:50 +08:00
SevenAddSix
5eab9719cb
HDFS-16480. Fix typo: indicies -> indices (#4020) 2022-02-28 21:12:18 +08:00
Hui Fei
b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998)" (#4038) 2022-02-28 21:05:19 +08:00
litao
8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998) 2022-02-28 21:03:59 +08:00
litao
6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 2022-02-25 01:23:20 +09:00
Viraj Jasani
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
jianghuazhu
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Chentao Yu
19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
litao
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 13:40:46 +09:00
secfree
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847) 2022-02-09 10:07:05 +08:00
singer-bin
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 2022-02-06 13:05:57 +08:00
litao
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web (#3943) 2022-01-30 16:13:58 +08:00
secfree
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 17:04:11 +09:00
Li MingXiang
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00
litao
c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode (#3831) 2022-01-26 17:24:09 +09:00
Bryan Beaudreault
94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 2022-01-25 11:42:35 +00:00
Thinker313
43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-25 15:26:18 +08:00
daimin
5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-01-24 12:04:26 +05:30
litao
15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. (#3839). Contributed by tomscut. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-24 14:26:30 +08:00
qinyuren
db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages (#3883) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-19 17:59:42 +09:00
litao
e355646330
HDFS-16399. Reconfig cache report parameters for datanode (#3841) 2022-01-19 15:10:39 +09:00
qinyuren
fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 2022-01-19 13:42:33 +09:00
Jackson Wang
d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 22:38:11 +09:00
litao
f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 14:48:10 +09:00
Xiangyi Zhu
034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063). Contributed by Xiangyi Zhu. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-13 12:56:19 +08:00
litao
46b02788bd
HDFS-16371. Exclude slow disks when choosing volume (#3753) 2022-01-06 09:31:00 +09:00
Ashutosh Gupta
9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 13:25:16 +09:00
Symious
b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline (#3704) 2021-12-30 00:27:11 +09:00
jianghuazhu
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 17:24:12 +09:00
KevinWikant
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 22:59:08 +09:00
secfree
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
secfree
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
liubingxing
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-17 13:31:28 +08:00
Ayush Saxena
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:21:51 +09:00
litao
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 2021-12-14 11:19:06 +09:00