big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,937 Commits 2 Branches 2 Tags 264 MiB
e5eb404bb3
Commit Graph

9990 Commits

Author SHA1 Message Date
Viraj Jasani
1263e024b9
HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 14:26:52 -08:00
Simbarashe Dzinamarira
f26d8bc9bd
HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 19:46:51 +00:00
huhaiyang
e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-01-11 09:12:26 +08:00
Chengbing Liu
4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-01-10 10:03:25 -08:00
huhaiyang
f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Simbarashe Dzinamarira
b56d483258
HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. (#5282) 2023-01-09 23:55:13 +00:00
Simbarashe Dzinamarira
cd19da1309
HDFS-16877: Enables state context for namenode in TestObserverWithRouter (#5257) 2023-01-07 00:18:35 +00:00
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
Yubi Lee
4511c360b9
HDFS-16883. Duplicate field name in hdfs-default.xml (#5271). Contributed by YUBI LEE. 2023-01-05 04:58:43 +05:30
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
Neil
d25c1be517
HDFS-16861. RBF. Truncate API always fails when dirs use AllResolver oder on Router (#5184) 
Co-authored-by: xiezhineng <xiezhineng@corp.netease.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-26 21:19:33 +08:00
ZanderXu
df093ef9af
HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time (#5098) 2022-12-23 14:16:42 +08:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
Daniel-009497
7ff326129d
HDFS-16871 DiskBalancer process may throw IllegalArgumentException when the target DataNode has capital letter in hostname (#5240) 2022-12-21 07:12:02 +08:00
Xing Lin
f7bdf6c667
HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down (#5160) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-12-16 08:46:14 -08:00
Happy-shi
c5b42d59d2
HDFS-16866. Fix a typo in Dispatcher (#5202) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-16 11:07:41 +08:00
curie71
fdcbc8b072
HDFS-16868. Fix audit log duplicate issue when an ACE occurs in FSNamesystem. (#5206). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-12-13 12:24:51 +08:00
Anurag P
e76616f690
HDFS-16860 Upgrade moment.min.js to 2.29.4 (#5194) 2022-12-09 11:18:44 +05:30
dingshun3016
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. (#5180) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-09 08:10:04 +08:00
ZanderXu
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces (#5123) 2022-12-05 16:15:47 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. (#5050) 2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory (#4209) 2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy (#5128) 
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-30 09:43:15 +08:00
Owen O'Malley
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. (#5155) 2022-11-29 22:12:35 +00:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration (#5143) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-11-29 10:51:21 +09:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. (#5142) 2022-11-29 00:49:10 +00:00
Simbarashe Dzinamarira
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. (#5145) 2022-11-29 00:47:01 +00:00
ZanderXu
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md (#4922) 2022-11-25 09:26:28 +08:00
ZanderXu
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request (#5086) 2022-11-25 09:23:33 +08:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand (#5137) 2022-11-24 09:17:27 +08:00
ZanderXu
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap (#5125) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:07:27 +08:00
huhaiyang
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB (#5129) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-24 09:00:27 +08:00
huhaiyang
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate (#5063) 2022-11-23 09:50:57 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state (#4201) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory (#5099) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
 2022-11-21 08:26:16 -08:00
Owen O'Malley
c71a68ca80
HDFS-16844: Adds resilancy when StateStore gets exceptions. (#5138) 
Allows the StateStore to stay up when there are errors reading the data.
 2022-11-18 17:24:08 +00:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized (#5135) 
Co-authored-by: Lei Yang <leyang@linkedin.com>
 2022-11-15 23:06:37 +00:00
ZanderXu
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. (#4945). Contributed by ZanderXu. 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-11-13 22:01:35 +08:00
PJ Fanning
d340c4a7a1
HADOOP-18496. Upgrade okhttp3 and dependencies due to kotlin CVEs (#5035) 
Updates okhttp3 and okio so their transitive dependency on Kotlin
stdlib is free from recent CVEs.

okhttp3:okhttp => 4.10.0
okio:okio => 3.2.0
kotlin stdlib => 1.6.20

kotlin CVEs fixed:
 CVE-2022-24329
 CVE-2020-29582
 
Contributed by PJ Fanning.
 2022-11-12 14:14:19 +00:00
Simbarashe Dzinamarira
552ee44eba
HDFS-16834: Removes request stateID consistency constraint between clients in different connection pools. (#5121) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
 2022-11-11 15:26:31 +08:00
Simbarashe Dzinamarira
44b8bb7224
HDFS-16821: Fixes regression in HDFS-13522 that enables observer reads by default (#5078) 2022-11-07 08:43:04 -08:00
Takanobu Asanuma
660530205e
HDFS-16833. NameNode should log internal EC blocks instead of the EC block group when it receives block reports. (#5106) 
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-11-07 10:49:48 +09:00
huhaiyang
e9319e696c
HDFS-16811. Support DecommissionBackoffMonitor Parameters reconfigurablereconfigurable (#5068) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-11-04 14:18:59 +08:00
jianghuazhu
88f7f5bc01
HDFS-16802.Print options when accessing ClientProtocol#rename2(). (#5013) 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
 2022-10-28 16:31:20 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:03:15 +08:00
huhaiyang
d26c35b228
HDFS-16817. Remove useless DataNode lock related configuration (#5072) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: ZanderXu <zanderxu@apache.org>
 2022-10-27 17:28:19 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-10-27 14:39:01 +09:00
jianghuazhu
c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. (#5031) 2022-10-20 10:58:23 +08:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (#4750) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-17 18:56:15 +08:00
Ashutosh Gupta
d6b1e1eeb6
HDFS-16777. datatables@1.10.17 sonatype-2020-0988 vulnerability (#5003) 2022-10-12 14:51:12 -07:00
huhaiyang
d14b88c698
HDFS-16774.Improve async delete replica on datanode (#4903) 
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException

Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-11 16:22:56 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers (#4979) 
Contributed by P J Fanning
 2022-10-10 18:56:35 +01:00
ZanderXu
62ff4e36cf
HDFS-16787. Remove the redundant lock in DataSetLockManager#removeLock (#4948). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 20:05:02 +08:00
ZanderXu
b0b2cb4a16
HDFS-16783. Remove the redundant lock in deepCopyReplica and getFinalizedBlocks (#4942). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 19:52:22 +08:00
ZanderXu
b0bfd09c41
HDFS-16798. HDFS-16798. SerialNumberMap should decrease current counter if the item exist. (#4987). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-09 18:58:09 +08:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917) 
Contributed-by: navinko <nakumr@cloudera.com>
 2022-09-30 23:28:12 +08:00
Ashutosh Gupta
d9f435f6ac
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source (#4886) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-27 15:44:25 +09:00
Kidd5368
9a29075f91
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901) 2022-09-23 12:27:56 +09:00
Samrat
740e1ef357
HDFS-16706. ViewFS doc points to wrong mount table name (#4803) 2022-09-21 16:55:20 +05:30
GuoPhilipse
40ab9c8ba7
HDFS-16341. Fix BlockPlacementPolicy details in hdfs defaults. (#3691). Contributed by guophilipse. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:16:24 +05:30
ZanderXu
e68006cd70
HDFS-16772. refreshHostsReader should use the latest configuration (#4890) 2022-09-19 13:27:07 -07:00
ZanderXu
a73c4804d8
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs (#4871) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-16 10:09:01 -07:00
ZanderXu
43c1ebae16
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException (#4882) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-15 12:44:36 -07:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
Renukaprasad C
8ce71c8882
[Documentation] RBF: Duplicate statement to be removed for better readabilty (#4881) 2022-09-13 09:24:05 -07:00
caozhiqiang
1923096adb
Allow block reconstruction pending timeout to be refreshable (#4567) 
Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-12 11:45:01 +09:00
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients. 
Fixes #4311

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-09 15:43:28 -07:00
ZanderXu
4a01fadb94
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-09 12:47:36 +05:30
ZanderXu
c947c326e8
HDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger than HighestWrittenTxId (#4560) 
Co-authored-by: Zander Xu <zengqiang.xu@shopee.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-09-06 10:12:55 -07:00
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF (#4813). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-05 17:46:15 +05:30
ZanderXu
ac42519ade
HDFS-16593. Correct the BlocksRemoved metric on DataNode side (#4353). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-09-05 19:35:48 +08:00
ZanderXu
7b239a80fe
HDFS-16750. NameNode should use NameNode.getRemoteUser() to log audit event to avoid possible NPE (#4821) 2022-09-02 10:23:03 -07:00
Steve Vaughan
2dd8b1342e
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution (#4833) 
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.

Contributed by Steve Vaughan Jr
 2022-09-01 14:00:15 +01:00
zhangshuyan0
71778a6cc5
HDFS-16735. Reduce the number of HeartbeatManager loops. (#4780). Contributed by Shuyan Zhang. 
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-29 11:30:21 +08:00
ZanderXu
5567154f71
HDFS-16734. RBF: fix some bugs when handling getContentSummary RPC (#4763) 2022-08-26 16:04:33 -07:00
Gautham B A
5736b34b2a
HDFS-16736. Link to Boost library in libhdfspp (#4782) 2022-08-26 09:11:44 -07:00
zhengchenyu
231a4468cd
HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed (#4756) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-08-25 10:37:25 -07:00
ZanderXu
1691cccc89
HDFS-16738. Invalid CallerContext caused NullPointerException (#4791) 2022-08-25 17:12:27 +08:00
ZanderXu
8d4f51c432
HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices (#4734). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-24 20:27:15 +05:30
ZanderXu
c37f01d95b
HDFS-16724. RBF should support get the information about ancestor mount points (#4719) 2022-08-23 13:25:42 -07:00
Simba Dzinamarira
a3b1bafa34
HDFS-16669: Enhance client protocol to propagate last seen state IDs for multiple nameservices. 
Fixes #4584

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-08-23 11:12:50 -07:00
Steve Vaughan
6fbc38db95
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724) 2022-08-23 08:03:57 -07:00
ZanderXu
183f09b1da
HDFS-16717. Replace NPE with IOException in DataNode.class (#4699). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-23 18:17:32 +08:00
Steve Vaughan
a9e5fb3313
HDFS-16684. Exclude the current JournalNode (#4723) 
Exclude bound local addresses, including the use of a wildcard address in the bound host configurations.
* Allow sync attempts with unresolved addresses
* Update the comments.
* Remove unused import

Signed-off-by: stack <stack@apache.org>
 2022-08-22 09:52:45 -07:00
jianghuazhu
7f176d080c
HDFS-16729. RBF: fix some unreasonably annotated docs. (#4745) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-21 07:29:31 +09:00
Ashutosh Gupta
5cc8c574d1
HDFS-16676. DatanodeAdminManager$Monitor reports a node as invalid continuously (#4626) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-08-18 02:25:09 +08:00
Ashutosh Gupta
86abeb401e
HDFS-16730. Update the doc that append to EC files is supported (#4749) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-08-17 05:41:41 +08:00
xuzq
622ca0d51f
HDFS-16705. RBF supports healthMonitor timeout configurable and caching NN and client proxy in NamenodeHeartbeatService (#4662) 2022-08-15 13:55:16 -07:00
xuzq
b1d4af2492
HDFS-16704. Datanode return empty response instead of NPE for GetVolumeInfo during restarting (#4661). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-15 20:57:27 +08:00
Hui Fei
86cc96c493
Revert "HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628)" (#4743) 2022-08-15 19:48:05 +08:00
xuzq
e0c8c6eed4
HDFS-16678. RBF should supports disable getNodeUsage() in RBFMetrics (#4606) 2022-08-12 12:01:58 -07:00
xuzq
521e65acfe
HDFS-16723. Replace incorrect SafeModeException with StandbyException in RouterRpcServer.class (#4716). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-13 00:21:11 +05:30
xuzq
59619ad247
HDFS-16689. NameNode may crash when transitioning to Active with in-progress tailer if there are some abnormal JNs. (#4628) 2022-08-12 12:19:28 +08:00
Steve Vaughan
2005582a28
HDFS-16702. MiniDFSCluster should report cause of exception in assert error (#4680) 
When the MiniDFSClsuter detects that an exception caused an exit, it should include that exception as the cause for the AssertionError that it throws. The current AssertError simply reports the message "Test resulted in an unexpected exit" and provides a stack trace to the location of the check for an exit exception. This patch adds the original exception as the cause of the AssertError.
 2022-08-11 13:52:39 -07:00
xuzq
09cabaad68
HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets (#4531) 2022-08-11 13:23:32 -07:00
xuzq
895f7c51fd
HDFS-16709. Remove redundant cast in FSEditLogOp.class (#4667). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-09 00:54:01 +05:30
xuzq
25ccdc77af
HDFS-16648. Add isDebugEnabled check for debug blockLogs in some classes (#4529) 2022-08-06 21:34:01 +08:00
Steve Loughran
62dbefd8f2
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files 
Add the r3.3.4 changelog, release notes and jdiff xml files.
 2022-08-05 14:06:22 +01:00
zhangshuyan0
dbf73e16b1
HADOOP-18364. All method metrics related to the RPC protocol should be initialized. (#4624). Contributed by Shuyan Zhang. 
Reviewed-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-08-04 13:00:37 +08:00
xuzq
8eebf40b1a
HDFS-16642. Moving the selecting inputstream from journalnode in EditLogTailer outof FSNLock (#4497) 2022-08-04 11:04:28 +08:00
Steve Vaughan
0fc7dd8228
HDFS-16687. RouterFsckServlet replicates code from DfsServlet base class (#4681) 2022-08-03 09:14:11 -07:00
xuzq
0f36539d60
HDFS-16712. Fix incorrect placeholder in DataNode.java (#4672). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-08-03 13:01:41 +05:30
Ashutosh Gupta
69f6fdb757
HADOOP-18301. Upgrade commons-io to 2.11.0 (#4455) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-08-03 10:44:39 +09:00
xuzq
f80fab2b90
HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout (#4597) 2022-07-28 15:42:53 +08:00
xuzq
a5adc27c99
HDFS-16658. Change logLevel from DEBUG to INFO if logEveryBlock is true (#4559). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-07-28 15:30:22 +08:00
slfan1989
06ac327e88
HDFS-16619. Fix HttpHeaders.Values And HttpHeaders.Names Deprecated Import (#4406) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-07-28 11:07:24 +08:00
xuzq
24560f2eb5
HDFS-16660. Improve Code With Lambda in IPCLoggerChannel class (#4561) 2022-07-27 18:53:05 -07:00
xuzq
01a2e0f6bd
HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. (#4155) 
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-07-26 04:30:00 +08:00
Gautham B A
6ba2c53720
HDFS-16681. Do not pass GCC flags for MSVC in libhdfspp (#4615) 
* This PR ensures that the GCC flag
  -Wno-missing-field-initializers isn't passed
  for MSVC.
 2022-07-25 22:55:38 +05:30
Neil
2f49eec5dd
HDFS-16655. OIV: print out erasure coding policy name in oiv Delimited output (#4541). Contributed by Max Xie. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-07-25 17:39:25 +08:00
Gautham B A
8f83d9f56d
HDFS-16680. Skip libhdfspp Valgrind tests on Windows (#4611) 
* The CMake test libhdfs_mini_stress_valgrind
  requires Valgrind.
* This PR skips this test on Windows since
  Valgrind isn't available.
 2022-07-23 23:22:13 +05:30
Gautham B A
7de9b5ee27
HDFS-16467. Ensure Protobuf generated headers are included first (#4601) 
* This PR ensures that the Protobuf generated headers
  are always included first, even when these headers
  are included transitively.
* This problem is specific to Windows only.
 2022-07-23 23:20:15 +05:30
xuzq
2c96357051
HDFS-15079. RBF: Namenode needs to use the actual client Id and callId when going through RBF proxy. (#4530) 2022-07-23 22:19:37 +08:00
Masatake Iwasaki
3cce41a1f6 Make upstream aware of 3.2.4 release. 
(cherry picked from commit e1637a57dfd41385dbce5de90620c48a45abb263)
 2022-07-22 02:27:19 +00:00
slfan1989
2f6916a313
HDFS-16605. Improve Code With Lambda in hadoop-hdfs-rbf moudle. (#4375) 2022-07-21 18:42:55 -07:00
wangzhaohui
08a940d5dd
HDFS-16640. RBF: Show datanode IP list when click DN histogram in Router (#4488) 2022-07-21 16:21:31 -07:00
Gautham B A
d07256a96d
HDFS-16667. Use malloc for buffer allocation in uriparser2 (#4576) 
* Windows doesn't support variables for specifying
  the array size.
* This PR uses malloc to fix this issue.
 2022-07-20 21:57:28 +05:30
Gautham B A
6415eb04e8
HDFS-16665. Fix duplicate sources for HDFS test (#4573) 
* The library target hdfspp_test_shim_static is
  built using the following sources, which
  causes duplicate symbols to be defined -
  - hdfs_shim.c
  - ${LIBHDFSPP_BINDING_C}/hdfs.cc
* ${LIBHDFSPP_BINDING_C}/hdfs.cc is redundant
  and removing this fixes the issue.
 2022-07-19 21:39:06 +05:30
Gautham B A
4fb799e6c5
HDFS-16464. Create only libhdfspp static libraries for Windows (#4571) 
* Added the appropriate CMake flags and
  commands to enable only statically
  linked libraries and executables to
  be built on Windows.
 2022-07-19 21:37:22 +05:30
Gautham B A
21b8952125
HDFS-16666. Pass CMake args for Windows in pom.xml (#4574) 
* This PR passes the necessary CMake args in the
  pom.xml needed for building HDFS native client
  on Windows.
* These arguments are exposed as maven options
  and can be passed from the command-line.
 2022-07-19 10:45:59 +05:30
Gautham B A
440f4c2b28
HDFS-16654. Link OpenSSL lib for CMake deps check (#4538) 
* The check_c_source_compiles fails on Windows
  while linking with an "unable to resolve
  external symbol" error.
* This PR links OpenSSL lib for this check to
  fix this issue.
 2022-07-17 20:47:30 +05:30
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits (#4252) 2022-07-16 04:52:12 +08:00
Murali Krishna
2835174a4c
HDFS-16652. Upgrade jquery datatable version references to v1.10.19 (#4562) 2022-07-14 18:27:07 +05:30
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC (#4524). Contributed by ZanderXu. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-14 07:26:40 +05:30
Gautham B A
8e39e35bea
HDFS-16466. Implement Linux permission flags on Windows (#4526) 
* HDFS-16466. Implement Linux permission flags on Windows

* statinfo.cc uses POSIX permission flags.
  These flags aren't available for Windows.
* This PR implements the equivalent flags
  on Windows to make this cross platform
  compatible.
 2022-07-08 09:29:13 +05:30
cxzl25
ea46f49b04
HDFS-16638. Add isDebugEnabled check for debug blockLogs in BlockManager. (#4480). Contributed by dzcxzl. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-04 02:36:29 +05:30
jianghuazhu
a333674785
HDFS-16647. Delete unused NameNode#FS_HDFS_IMPL_KEY. (#4525). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-07-03 01:47:02 +05:30
slfan1989
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' (#4422). Contributed by fanshilun. 2022-06-29 15:20:41 +05:30
jianghuazhu
321a4844ad
HADOOP-18314. Add some description for PowerShellFencer. (#4505) 2022-06-28 19:06:39 -07:00
Ashutosh Gupta
b7edc6c60c
HDFS-16633. Fixing when Reserved Space For Replicas is not released on some cases (#4452) 
* HDFS-16633.Reserved Space For Replicas is not released on some cases

Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-24 18:35:00 +05:30
Samrat
e8fd914c58
HDFS-16616. remove use of org.apache.hadoop.util.Sets (#4400) 
Co-Authored by: Samrat Deb
 2022-06-22 10:17:36 +05:30
Viraj Jasani
7a1d811197
HDFS-16637. TestHDFSCLI#testAll consistently failing (#4466). Contributed by Viraj Jasani. 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-21 13:42:43 +05:30
Ashutosh Gupta
efc2761d32
HDFS-16635.Fixed javadoc error in Java 11 (#4451) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-06-20 11:14:31 +05:30
KevinWikant
cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-20 11:20:24 +09:00
Viraj Jasani
cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-20 09:21:00 +08:00
Viraj Jasani
e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-18 12:17:20 +08:00
xuzq
4893f00395
HDFS-16600. Fix deadlock of fine-grain lock for FsDatastImpl of DataNode. (#4367). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-17 22:05:33 +08:00
caozhiqiang
9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu
6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-16 19:18:58 +08:00
章锡平
f8c7e67fdc
HDFS-16628 RBF: Correct target directory when move to trash for kerberos login user. (#4424). Contributed by Xiping Zhang. 2022-06-15 21:16:24 +08:00
Gautham B A
5a40224b53
HDFS-16469. Locate protoc-gen-hrpc across platforms (#4434) 
* We use the TARGET_FILE CMake
  generator expression to get
  the location of the
  protoc-gen-hrpc CMake target.
 2022-06-15 15:28:10 +05:30
xuzq
d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-06-14 21:48:03 +08:00
Steve Vaughan
bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414) 
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
 2022-06-13 23:42:18 -07:00
slfan1989
170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989
02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq
af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-06-10 12:00:56 -07:00
jianghuazhu
7f5a34dfaa
HDFS-16621.Remove unused JNStorage#getCurrentDir() (#4404). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-06-10 12:12:41 +05:30
Gautham B A
d557c44d94
HDFS-16463. Make dirent cross platform compatible (#4370) 
* jnihelper.c in HDFS native client uses
  dirent.h. This header file isn't available
  on Windows.
* This PR provides a cross platform
  compatible implementation for dirent
  under the XPlatform library.
 2022-06-10 09:59:07 +05:30
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-09 22:39:00 +09:00
slfan1989
2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-09 09:34:25 +08:00
Viraj Jasani
0a65883c50
HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 16:53:45 +08:00
Samrat
7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-06 17:58:49 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-06-04 06:10:14 +08:00
Gautham B A
a406f49bc0
HDFS-16602. Use "defined" directive along with #if (#4371) 
* The #if directive expects a
  boolean expression. This PR
  uses the more appropriate
  "defined" directive.
 2022-06-03 22:05:20 +05:30
Stephen O'Donnell
34a973a90e
HDFS-16610. Make fsck read timeout configurable (#4384) 2022-06-01 20:36:01 +01:00
Gautham B A
4496e2daf6
HDFS-16604. Install gtest via FetchContent_Declare in CMake (#4374) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-06-01 13:45:44 +09:00
slfan1989
e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372) 
Co-authored-by: slfan1989 <louj1988@@>
 2022-06-01 06:45:07 +08:00
Masatake Iwasaki
3228142f53 Make upstream aware of 2.10.2 release. 
(cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0)

Conflicts:
	hadoop-project-dist/pom.xml
 2022-06-01 00:53:01 +09:00
slfan1989
5952934ad5
HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-31 01:59:17 +05:30
slfan1989
995f7b0956
HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362). Contributed by fanshilun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:48:34 +05:30
xuzq
dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-05-28 09:43:37 +05:30
Ayush Saxena
96985f4c45
HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-05-27 09:06:07 +05:30
Ashutosh Gupta
bee538f785
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-27 05:18:15 +08:00
Stephen O'Donnell
297f0f6d6a
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-05-27 01:11:10 +08:00
Rishabh Sharma
0be1fde962
HDFS-16561. Handle error returned by strtol 
* strtol is used in hdfs-chmod.cc. The call
  to strtol could error out when an invalid
  input is provided.
* This PR handles the error given out by
   strtol.
 2022-05-26 11:09:49 +05:30
slfan1989
f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335) 
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2022-05-24 23:11:10 +05:30
Michael Stack
5ab747222b
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338) 
* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
 2022-05-23 08:45:08 -07:00
jianghuazhu
4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu. 
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-05-22 21:59:38 +08:00
Viraj Jasani
93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-21 09:28:59 +08:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 03:21:13 +09:00
Ashutosh Gupta
fb910bd906
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-21 02:53:14 +09:00
Steve Loughran
6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files. 
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
 2022-05-17 19:00:54 +01:00
Hui Fei
dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen
680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
singer-bin
d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Gautham B A
1350539f2d
HDFS-16465. Remove redundant strings.h inclusions (#4279) 
* Remove redundant strings.h inclusions

* strings.h was included in a bunch of
  C/C++ files and were redundant.
* Also, strings.h is not available on
  Windows and thus isn't cross-platform
  compatible.

* Build for all platforms in CI

* Revert "Build for all platforms in CI"

This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb.

* Debug failure on Centos 8

* Skipping pipeline run on
  Centos 7 to debug the
  failure on Centos 8.

* Revert "Debug failure on Centos 8"

This reverts commit e365e34d6fab9df88f4df622910ddb28a8c8796f.
 2022-05-11 23:04:22 +05:30
Viraj Jasani
6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
daimin
d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
Ashutosh Gupta
01027e52a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:05:34 +09:00
Ashutosh Gupta
ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
daimin
29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Gautham B A
d346be9bd2
HDFS-16564. Use uint32_t for hdfs_find (#4245) 
* hdfs_find uses u_int32_t type for
  storing the value for the max-depth
  command line argument.
* The type u_int32_t isn't standard,
  isn't available on Windows and thus
  breaks cross-platform compatibility.
  We need to replace this with uint32_t
  which is available on all platforms since
  it's part of the C++ standard.
 2022-05-04 20:19:08 +05:30
Murali Krishna
9807dfd25c
HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240) 2022-05-03 17:42:45 +05:30
Viraj Jasani
2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
Gautham B A
88155cebe9
HDFS-16468. Define ssize_t for Windows (#4228) 
* Some C/C++ files use ssize_t data type.
  This isn't available for Windows and we
  need to define an alias for this and set it
  to an appropriate type to make it cross
  platform compatible.
 2022-04-29 22:25:09 +05:30
smarthan
7bd7725532
HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211). Contributed by Chengwei Wang. 2022-04-29 22:59:23 +08:00
Viraj Jasani
b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack
9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
 2022-04-28 10:28:01 -07:00
stack
4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)" 
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
 2022-04-28 10:26:46 -07:00
huaxiangsun
bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
 2022-04-28 10:24:37 -07:00
Felix Nguyen
b4ff49a394
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168) 2022-04-27 14:42:30 +08:00
smarthan
f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
litao
f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Ashutosh Gupta
89279f0b90
HDFS-16255. RBF: Fix dead link to fedbalance document (#4193) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:11:41 +09:00
Hui Fei
17d64ba495
Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232) 2022-04-24 17:18:31 +08:00
litao
acc0e0a210
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035) 2022-04-24 14:54:41 +08:00
daimin
aebd55f788
HDFS-16519. Add throttler to EC reconstruction (#4101) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-04-23 12:45:22 +09:00
litao
5ebbacc480
HDFS-16552. Fix NPE for TestBlockManager (#4210) 2022-04-23 12:14:06 +09:00
smarthan
bf4730b4d6
HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. (#4061). Contributed by Chengwei Wang. 2022-04-21 12:32:12 +08:00
S O'Donnell
a4683be65e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit dbeeee0363.
 2022-04-20 20:34:43 +01:00
qinyuren
76bbd17374
HDFS-16544. EC decoding failed due to invalid buffer (#4179) 2022-04-20 15:04:27 +09:00
Ashutosh Gupta
ec0ff1dc04
HDFS-16035. Remove DummyGroupMapping as it is not longer used anywhere. (#4183) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-19 14:35:23 +09:00
qinyuren
52e152f8b0
HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
 2022-04-19 13:37:28 +09:00
Happy-shi
f74f241636
HDFS-16541. Fix a typo in NameNodeLayoutVersion (#4176) 2022-04-18 22:53:57 +05:30
jianghuazhu
900682e712
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 11:21:51 +08:00
Quanlong Huang
35d4c02bcc
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158) 
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
 2022-04-18 10:32:29 +08:00
Stephen O'Donnell
dbeeee0363
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 2022-04-17 20:05:11 +08:00
He Xiaoqiao
cb975c3df6
HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141) Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-04-17 19:21:27 +08:00
Renukaprasad C
f14f305051
HDFS-16526. Add metrics for slow DataNode (#4162) 2022-04-15 21:37:05 +05:30
caozhiqiang
cee8c62498
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) 2022-04-14 18:42:39 +09:00
daimin
c65c383b7e
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 2022-04-14 11:07:06 +08:00
Takanobu Asanuma
2efab92959
HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 2022-04-14 11:23:38 +09:00
Steve Loughran
a7b4e8f03e
HDFS-14478. Add libhdfs APIs for openFile (#4166) 
Contributed by Sahil Takiar
 2022-04-13 14:15:27 +01:00
qinyuren
45394433a1
HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 2022-04-13 11:27:37 +09:00
Gautham B A
e89fd9645b
HDFS-16474. Make HDFS tail tool cross platform (#4157) 
* The source files for hdfs_tail
  uses getopt for parsing the
  command line arguments.
* getopt is available only on
  Linux and thus, isn't cross
  platform.
* We need to replace getopt
  with boost::program_options
  to make these tools cross
  platform.
 2022-04-12 20:31:24 +05:30
Gautham B A
d5e97fe4d6
HDFS-16473. Make HDFS stat tool cross platform (#4145) 
* The source files for hdfs_stat
  uses getopt for parsing the
  command line arguments.
* getopt is available only on
  Linux and thus, isn't cross platform.
* We need to replace getopt with
  boost::program_options to make
  this tool cross platform.
 2022-04-08 23:06:39 +05:30
Takanobu Asanuma
37650ced81 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 2022-04-08 18:38:40 +09:00
litao
5412fbf6d4
HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001) 2022-04-08 12:26:39 +08:00
singer-bin
807a428b55
HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069) 2022-04-08 10:27:35 +09:00
Viraj Jasani
f70935522b
HADOOP-18188. Support touch command for directory (#4135) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-07 17:29:45 +09:00
Viraj Jasani
7c20602b17
HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 18:17:02 +09:00
wangzhaohui
61bbdfd3a7
HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-06 17:29:43 +09:00
Stephen O'Donnell
bbfe3500cf
HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 2022-04-06 08:49:52 +01:00
Gautham B A
4ef1d3eef9
HDFS-16472. Make HDFS setrep tool cross platform (#4130) 
* The source files for hdfs_setrep
   uses getopt for parsing the
   command line arguments.
* getopt is available only on Linux
   and thus, isn't cross platform.
* We need to replace getopt
  with boost::program_options
  to make this tool cross platform.
 2022-04-05 22:59:11 +05:30
litao
34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-04-03 01:36:03 +05:30
Xing Lin
15a5ea2c95
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094) 
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
 2022-03-31 21:59:09 +00:00
litao
9a4dddd640
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:01:48 -07:00
He Xiaoqiao
2bf78e2416
HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mode. (#4085). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-31 14:00:38 +08:00
litao
ac50657c37
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) 2022-03-31 00:51:17 +09:00
He Xiaoqiao
6eea28c3f3
HDFS-16498. Fix NPE for checkBlockReportLease #4057. Contributed by tomscut. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-30 14:56:04 +08:00
Lei Yang
a9b43966c0
HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:15:28 -07:00
Masatake Iwasaki
0fbd96a244 Make upstream aware of 3.2.3 release. 2022-03-28 08:07:50 +00:00
GuoPhilipse
046a6204b4
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-03-27 22:23:48 +09:00
litao
565e848d88
HDFS-16434. Add opname to read/write lock for remaining operations (#3915) 2022-03-25 20:19:01 +09:00
qinyuren
45ce1cce50
HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
 2022-03-23 15:03:17 +09:00
He Xiaoqiao
ef8bff0df9
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-23 11:32:12 +08:00
He Xiaoqiao
26ba3846cc
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang." 
This reverts commit 88975496d8.
 2022-03-23 11:01:35 +08:00
Gautham B A
81879eb9cb
HDFS-16471. Make HDFS ls tool cross platform (#4086) 
* The source files for hdfs_ls
   uses getopt for parsing the
   command line arguments.
* getopt is available only on
   Linux and thus, isn't cross
   platform.
* Thus, we need to replace
   getopt with
   boost::program_options to
   make this tool cross platform.
 2022-03-22 23:12:00 +05:30
He Xiaoqiao
88975496d8
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-22 22:28:09 +08:00
Owen O'Malley
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 09:27:35 -07:00
litao
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-03-21 12:40:51 +05:30
He Xiaoqiao
62d59e516e
HDFS-16504. Add parameter for NameNode to process getBloks request. (#4068). Contributed by Max Xie. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-20 14:01:45 +08:00
Gautham B A
a631f45a99
HDFS-16470. Make HDFS find tool cross platform (#4076) 
* The source files for hdfs_find uses
   getopt for parsing the command
   line arguments. getopt is available
   only on Linux and thus, isn't cross
   platform.
* Thus, we need to replace getopt
   with boost::program_options to
   make hdfs_find cross platform.
 2022-03-18 21:41:01 +05:30
litao
7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058) 2022-03-17 12:04:48 +08:00
He Xiaoqiao
a237526988
HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by JiangHua Zhu. 2022-03-16 20:43:38 +08:00
Viraj Jasani
1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-16 09:32:29 +08:00
Xing Lin
8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 11:29:48 -07:00
Owen O'Malley
7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
He Xiaoqiao
a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
 2022-03-12 18:40:09 +08:00