big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,047 Commits 2 Branches 2 Tags 264 MiB
2005582a28
Commit Graph

5335 Commits

Author SHA1 Message Date
Steve Vaughan
2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
xuzq
895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
zhangshuyan0
dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
slfan1989
06ac327e88
HDFS-16619. Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Import (#4406) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-07-28 11:07:24 +08:00
xuzq
01a2e0f6bd
HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. (#4155) 
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-07-26 04:30:00 +08:00
Neil
2f49eec5dd
HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output (#4541). Contributed by Max Xie. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-07-25 17:39:25 +08:00
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) 2022-07-16 04:52:12 +08:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Ashutosh Gupta
b7edc6c60c
HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-24 18:35:00 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
Viraj Jasani
7a1d811197
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:42:43 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00
caozhiqiang
9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu
6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:18:58 +08:00
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
jianghuazhu
4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-22 21:59:38 +08:00
Viraj Jasani
93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:28:59 +08:00
singer-bin
d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Viraj Jasani
6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
daimin
d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
daimin
29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
Viraj Jasani
b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack
9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
 2022-04-28 10:28:01 -07:00
stack
4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)" 
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
 2022-04-28 10:26:46 -07:00
huaxiangsun
bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
 2022-04-28 10:24:37 -07:00
smarthan
f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
litao
f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Hui Fei
17d64ba495
Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232) 2022-04-24 17:18:31 +08:00
litao
acc0e0a210
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035) 2022-04-24 14:54:41 +08:00
litao
5ebbacc480
HDFS-16552. Fix NPE for TestBlockManager (#4210) 2022-04-23 12:14:06 +09:00
S O'Donnell
a4683be65e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit dbeeee0363.
 2022-04-20 20:34:43 +01:00
qinyuren
76bbd17374
HDFS-16544. EC decoding failed due to invalid buffer (#4179) 2022-04-20 15:04:27 +09:00
qinyuren
52e152f8b0
HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
 2022-04-19 13:37:28 +09:00
jianghuazhu
900682e712
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 11:21:51 +08:00
Quanlong Huang
35d4c02bcc
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
 2022-04-18 10:32:29 +08:00