big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,382 Commits 2 Branches 2 Tags 264 MiB
5022003e0f
Commit Graph

5370 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
dingshun3016
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. (#5180) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-09 08:10:04 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
huhaiyang
e9319e696c
HDFS-16811. Support DecommissionBackoffMonitor Parameters reconfigurablereconfigurable (#5068) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-04 14:18:59 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:03:15 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-10-27 14:39:01 +09:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (#4750) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-17 18:56:15 +08:00
huhaiyang
d14b88c698
HDFS-16774.Improve async delete replica on datanode (#4903) 
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException

Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-11 16:22:56 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers (#4979) 
Contributed by P J Fanning
 2022-10-10 18:56:35 +01:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917) 
Contributed-by: navinko <nakumr@cloudera.com>
 2022-09-30 23:28:12 +08:00
ZanderXu
e68006cd70
HDFS-16772. refreshHostsReader should use the latest configuration (#4890) 2022-09-19 13:27:07 -07:00
ZanderXu
a73c4804d8
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs (#4871) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-16 10:09:01 -07:00
ZanderXu
43c1ebae16
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException (#4882) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-15 12:44:36 -07:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
caozhiqiang
1923096adb
Allow block reconstruction pending timeout to be refreshable (#4567) 
Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-12 11:45:01 +09:00
ZanderXu
c947c326e8
HDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger than HighestWrittenTxId (#4560) 
Co-authored-by: Zander Xu <zengqiang.xu@shopee.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-09-06 10:12:55 -07:00
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-05 17:46:15 +05:30
ZanderXu
ac42519ade
HDFS-16593. Correct the BlocksRemoved metric on DataNode side (#4353). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-09-05 19:35:48 +08:00
Steve Vaughan
2dd8b1342e
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833) 
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr
 2022-09-01 14:00:15 +01:00
zhengchenyu
231a4468cd
HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-08-25 10:37:25 -07:00
ZanderXu
1691cccc89
HDFS-16738. Invalid CallerContext caused NullPointerException (#4791) 2022-08-25 17:12:27 +08:00
Steve Vaughan
6fbc38db95
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724) 2022-08-23 08:03:57 -07:00
Steve Vaughan
a9e5fb3313
HDFS-16684. Exclude the current JournalNode (#4723) 
Exclude bound local addresses, including the use of a wildcard address in the bound host configurations.
* Allow sync attempts with unresolved addresses
* Update the comments.
* Remove unused import

Signed-off-by: stack <stack@apache.org>
 2022-08-22 09:52:45 -07:00
Hui Fei
86cc96c493
Revert "HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628)" (#4743) 2022-08-15 19:48:05 +08:00
xuzq
59619ad247
HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628) 2022-08-12 12:19:28 +08:00
Steve Vaughan
2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
xuzq
895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
zhangshuyan0
dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
slfan1989
06ac327e88
HDFS-16619. Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Import (#4406) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-07-28 11:07:24 +08:00
xuzq
01a2e0f6bd
HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. (#4155) 
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-07-26 04:30:00 +08:00
Neil
2f49eec5dd
HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output (#4541). Contributed by Max Xie. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-07-25 17:39:25 +08:00
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) 2022-07-16 04:52:12 +08:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
Ashutosh Gupta
b7edc6c60c
HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-24 18:35:00 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
Viraj Jasani
7a1d811197
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:42:43 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00