ZanderXu
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces ( #5123 )
2022-12-05 16:15:47 +08:00
dingshun3016
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. ( #5050 )
2022-12-05 16:34:51 +09:00
litao
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory ( #4209 )
2022-11-30 07:44:21 -08:00
Kidd5368
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy ( #5128 )
...
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
2022-11-30 09:43:15 +08:00
Owen O'Malley
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. ( #5155 )
2022-11-29 22:12:35 +00:00
caozhiqiang
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration ( #5143 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-11-29 10:51:21 +09:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. ( #5142 )
2022-11-29 00:49:10 +00:00
Simbarashe Dzinamarira
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. ( #5145 )
2022-11-29 00:47:01 +00:00
ZanderXu
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md ( #4922 )
2022-11-25 09:26:28 +08:00
ZanderXu
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request ( #5086 )
2022-11-25 09:23:33 +08:00
huhaiyang
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand ( #5137 )
2022-11-24 09:17:27 +08:00
ZanderXu
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap ( #5125 )
...
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:07:27 +08:00
huhaiyang
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB ( #5129 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:00:27 +08:00
huhaiyang
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate ( #5063 )
2022-11-23 09:50:57 +08:00
litao
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state ( #4201 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 10:14:07 -08:00
zhengchenyu
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory ( #5099 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 08:26:16 -08:00
Owen O'Malley
c71a68ca80
HDFS-16844: Adds resilancy when StateStore gets exceptions. ( #5138 )
...
Allows the StateStore to stay up when there are errors reading the data.
2022-11-18 17:24:08 +00:00
Lei Yang
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized ( #5135 )
...
Co-authored-by: Lei Yang <leyang@linkedin.com>
2022-11-15 23:06:37 +00:00
ZanderXu
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. ( #4945 ). Contributed by ZanderXu.
...
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-11-13 22:01:35 +08:00
PJ Fanning
d340c4a7a1
HADOOP-18496. Upgrade okhttp3 and dependencies due to kotlin CVEs ( #5035 )
...
Updates okhttp3 and okio so their transitive dependency on Kotlin
stdlib is free from recent CVEs.
okhttp3:okhttp => 4.10.0
okio:okio => 3.2.0
kotlin stdlib => 1.6.20
kotlin CVEs fixed:
CVE-2022-24329
CVE-2020-29582
Contributed by PJ Fanning.
2022-11-12 14:14:19 +00:00
Simbarashe Dzinamarira
552ee44eba
HDFS-16834: Removes request stateID consistency constraint between clients in different connection pools. ( #5121 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
2022-11-11 15:26:31 +08:00
Simbarashe Dzinamarira
44b8bb7224
HDFS-16821: Fixes regression in HDFS-13522 that enables observer reads by default ( #5078 )
2022-11-07 08:43:04 -08:00
Takanobu Asanuma
660530205e
HDFS-16833. NameNode should log internal EC blocks instead of the EC block group when it receives block reports. ( #5106 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
2022-11-07 10:49:48 +09:00
huhaiyang
e9319e696c
HDFS-16811. Support DecommissionBackoffMonitor Parameters reconfigurablereconfigurable ( #5068 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-04 14:18:59 +08:00
jianghuazhu
88f7f5bc01
HDFS-16802.Print options when accessing ClientProtocol#rename2(). ( #5013 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
2022-10-28 16:31:20 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block ( #4697 )
...
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-10-27 19:03:15 +08:00
huhaiyang
d26c35b228
HDFS-16817. Remove useless DataNode lock related configuration ( #5072 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: ZanderXu <zanderxu@apache.org>
2022-10-27 17:28:19 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. ( #5079 )
...
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
2022-10-27 14:39:01 +09:00
jianghuazhu
c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. ( #5031 )
2022-10-20 10:58:23 +08:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS ( #4750 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-10-17 18:56:15 +08:00
Ashutosh Gupta
d6b1e1eeb6
HDFS-16777. datatables@1.10.17 sonatype-2020-0988 vulnerability ( #5003 )
2022-10-12 14:51:12 -07:00
huhaiyang
d14b88c698
HDFS-16774.Improve async delete replica on datanode ( #4903 )
...
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException
Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-10-11 16:22:56 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers ( #4979 )
...
Contributed by P J Fanning
2022-10-10 18:56:35 +01:00
ZanderXu
62ff4e36cf
HDFS-16787. Remove the redundant lock in DataSetLockManager#removeLock ( #4948 ). Contributed by ZanderXu.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-10-10 20:05:02 +08:00
ZanderXu
b0b2cb4a16
HDFS-16783. Remove the redundant lock in deepCopyReplica and getFinalizedBlocks ( #4942 ). Contributed by ZanderXu.
...
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-10-10 19:52:22 +08:00
ZanderXu
b0bfd09c41
HDFS-16798. HDFS-16798. SerialNumberMap should decrease current counter if the item exist. ( #4987 ). Contributed by ZanderXu.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-10-09 18:58:09 +08:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider ( #4917 )
...
Contributed-by: navinko <nakumr@cloudera.com>
2022-09-30 23:28:12 +08:00
Ashutosh Gupta
d9f435f6ac
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source ( #4886 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-09-27 15:44:25 +09:00
Kidd5368
9a29075f91
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task ( #4901 )
2022-09-23 12:27:56 +09:00
Samrat
740e1ef357
HDFS-16706. ViewFS doc points to wrong mount table name ( #4803 )
2022-09-21 16:55:20 +05:30
GuoPhilipse
40ab9c8ba7
HDFS-16341. Fix BlockPlacementPolicy details in hdfs defaults. ( #3691 ). Contributed by guophilipse.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-09-20 23:16:24 +05:30
ZanderXu
e68006cd70
HDFS-16772. refreshHostsReader should use the latest configuration ( #4890 )
2022-09-19 13:27:07 -07:00
ZanderXu
a73c4804d8
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs ( #4871 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-09-16 10:09:01 -07:00
ZanderXu
43c1ebae16
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException ( #4882 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-09-15 12:44:36 -07:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation.
...
Fixes #4127
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-09-14 16:00:14 -07:00
Renukaprasad C
8ce71c8882
[Documentation] RBF: Duplicate statement to be removed for better readabilty ( #4881 )
2022-09-13 09:24:05 -07:00
caozhiqiang
1923096adb
Allow block reconstruction pending timeout to be refreshable ( #4567 )
...
Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-09-12 11:45:01 +09:00
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
...
Fixes #4311
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-09-09 15:43:28 -07:00
ZanderXu
4a01fadb94
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry ( #4853 ). Contributed by ZanderXu.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-09-09 12:47:36 +05:30
ZanderXu
c947c326e8
HDFS-16659. JournalNode should throw NewerTxnIdException when SinceTxId is bigger than HighestWrittenTxId ( #4560 )
...
Co-authored-by: Zander Xu <zengqiang.xu@shopee.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
2022-09-06 10:12:55 -07:00