LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure ( #6249 )
...
Contributed by liuguanghua.
2023-11-13 18:16:31 +00:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations ( #6198 )
...
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.
Contributed by Tom McCormick
2023-11-08 14:25:21 +00:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows ( #5943 )
2023-11-01 09:10:15 -07:00
Hiroaki Segawa
93a3c6e2cd
HDFS-17024. Potential data race introduced by HDFS-15865 ( #6223 )
2023-10-26 22:25:00 -07:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. ( #4996 )
...
Protobuf 2.5 JAR is no longer needed at runtime.
The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.
* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
...with a ShadedProtobufHelper in there which has shaded protobuf refs
only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
* The maven scope of the dependency can be set with the common.protobuf2.scope
option. It can be set to "provided" in a build:
-Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
over-complex. RouterAdminProtocolTranslatorPB is left alone for this
reason.
Contributed by Steve Loughran
2023-10-13 13:48:38 +01:00
PJ Fanning
c16484ffb2
HADOOP-18890. Remove use of okhttp in runtime code ( #6057 )
...
Contributed by PJ Fanning
2023-09-19 12:38:36 +01:00
hfutatzhanghb
18f9989ff2
HDFS-17192. Add bock info when constructing remote block reader meets IOException. ( #6081 ). Contributed by farmmamba.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-09-18 20:56:45 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS ( #5992 ) Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-09-02 15:24:52 +08:00
Benjamin Teke
43c889636a
YARN-11535. Remove jackson-dataformat-yaml dependency. ( #5970 )
2023-08-22 16:42:49 +02:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. ( #5941 ). Contributed by Shuyan Zhang.
...
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-08-16 20:29:19 +08:00
Xing Lin
80fefd093f
HDFS-17067 Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy ( #5803 )
2023-07-20 10:46:41 -07:00
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS ( #5836 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-07-14 11:53:36 +05:30
Xing Lin
ab47a658bd
HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy ( #5700 )
2023-06-14 10:53:56 -07:00
zhangshuyan
9c989515ba
HDFS-17037. Consider nonDfsUsed when running balancer. ( #5715 ). Contributed by Shuyan Zhang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-06-09 16:19:08 +08:00
zhtttylz
d9980ab40f
HDFS-17029. Support getECPolices API in WebHDFS ( #5698 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-06-05 17:33:37 +05:30
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module ( #5503 )"
...
This reverts commit 03a499821c
.
2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module ( #5503 )
...
Reviewed-by: Duo Zhang <zhangduo@apache.org>
2023-06-02 18:07:34 -07:00
Simbarashe Dzinamarira
d92a5815f4
HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. ( #5693 )
2023-05-31 10:20:19 -07:00
Keyao Li
339bc7b3a6
HDFS-16653. Improve error messages in ShortCircuitCache. ( #5568 ). Contributed by ECFuzz.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-19 07:43:18 +05:30
Xianming Lei
441fb23293
HDFS-17018. Improve dfsclient log format. ( #5668 ). Contributed by Xianming Lei.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-19 06:25:43 +05:30
zhtttylz
0c77629849
HDFS-17001. Support getStatus API in WebHDFS ( #5628 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-13 03:49:13 +05:30
Tak Lon (Stephen) Wu
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common ( #5553 )
...
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them. Applications which use the leasing APIs should migrate to the new
interface where possible.
Contributed by Stephen Wu
2023-05-03 11:05:55 +01:00
zhtttylz
c9e0af9961
HDFS-16981. Support getFileLinkStatus API in WebHDFS ( #5572 ). Contributed by Hualong Zhang.
...
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-25 03:30:56 +05:30
Melissa You
2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 ( #5536 )
2023-04-13 09:07:42 -07:00
zhtttylz
523ff81624
HDFS-16952. Support getLinkTarget API in WebHDFS ( #5517 )
...
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-04-06 19:44:47 +08:00
Viraj Jasani
0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem ( #5492 )
2023-03-21 10:23:16 +08:00
Tom
162288bc0a
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… ( #5322 )
...
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations.
ignoredNodes list is only used on hedged read codepath
Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
2023-03-01 19:47:04 +00:00
zhtttylz
a3b500d046
HDFS-16916. Improve the use of JUnit Test in DFSClient ( #5404 )
...
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-02-18 13:17:37 +08:00
Viraj Jasani
90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future ( #5315 )
2023-02-14 03:24:06 +08:00
zhtttylz
72b760130a
HDFS-16893. Standardize the usage of DFSClient debug log ( #5303 )
...
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2023-01-19 07:56:41 +08:00
Xing Lin
f7bdf6c667
HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down ( #5160 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
2022-12-16 08:46:14 -08:00
ZanderXu
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces ( #5123 )
2022-12-05 16:15:47 +08:00
Simbarashe Dzinamarira
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. ( #5142 )
2022-11-29 00:49:10 +00:00
PJ Fanning
d340c4a7a1
HADOOP-18496. Upgrade okhttp3 and dependencies due to kotlin CVEs ( #5035 )
...
Updates okhttp3 and okio so their transitive dependency on Kotlin
stdlib is free from recent CVEs.
okhttp3:okhttp => 4.10.0
okio:okio => 3.2.0
kotlin stdlib => 1.6.20
kotlin CVEs fixed:
CVE-2022-24329
CVE-2020-29582
Contributed by PJ Fanning.
2022-11-12 14:14:19 +00:00
Simbarashe Dzinamarira
44b8bb7224
HDFS-16821: Fixes regression in HDFS-13522 that enables observer reads by default ( #5078 )
2022-11-07 08:43:04 -08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block ( #4697 )
...
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-10-27 19:03:15 +08:00
jianghuazhu
c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. ( #5031 )
2022-10-20 10:58:23 +08:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS ( #4750 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-10-17 18:56:15 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers ( #4979 )
...
Contributed by P J Fanning
2022-10-10 18:56:35 +01:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider ( #4917 )
...
Contributed-by: navinko <nakumr@cloudera.com>
2022-09-30 23:28:12 +08:00
Ashutosh Gupta
d9f435f6ac
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source ( #4886 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-09-27 15:44:25 +09:00
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
...
Fixes #4311
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-09-09 15:43:28 -07:00
ZanderXu
be4c638e4c
HDFS-16748. RBF: DFSClient should uniquely identify writing files by namespace id and iNodeId via RBF ( #4813 ). Contributed by ZanderXu.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-09-05 17:46:15 +05:30
xuzq
01a2e0f6bd
HDFS-16533. COMPOSITE_CRC failed between replicated file and striped file due to invalid requested length. ( #4155 )
...
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-07-26 04:30:00 +08:00
RuinanGu
9376b65989
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits ( #4252 )
2022-07-16 04:52:12 +08:00
xuzq
6f9c4359ec
HDFS-16283. RBF: reducing the load of renewLease() RPC ( #4524 ). Contributed by ZanderXu.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-07-14 07:26:40 +05:30
Ashutosh Gupta
6e11c94170
HDFS-16576. Remove unused imports in HDFS project ( #4389 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-06-09 22:39:00 +09:00
Viraj Jasani
25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits ( #4357 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-06-04 06:10:14 +08:00
Stephen O'Donnell
34a973a90e
HDFS-16610. Make fsck read timeout configurable ( #4384 )
2022-06-01 20:36:01 +01:00
Viraj Jasani
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 ( #4267 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-21 03:21:13 +09:00