big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,429 Commits 2 Branches 2 Tags 264 MiB
49a495803a
Commit Graph

1096 Commits

Author SHA1 Message Date
Sadanand Shenoy
49a495803a
HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551) 
Contributed by Sadanand Shenoy
 2024-09-25 17:54:09 +01:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
 2024-09-24 16:50:22 +01:00
Steve Loughran
57e62ae07f
Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)" 
This reverts commit 6c01490f14.
 2024-09-05 14:35:50 +01:00
Syed Shameerur Rahman
6c01490f14
YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631) 
To support YARN deployments in clusters without HDFS
some changes have been made in packaging

* new hadoop-common class org.apache.hadoop.fs.HdfsCommonConstants
* hdfs class org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair moved
  from hdfs-client to hadoop-common
* YARN handlers for DSQuotaExceededException replaced by use of superclass
  ClusterStorageCapacityExceededException.  

Contributed by Syed Shameerur Rahman
 2024-09-04 13:26:42 +01:00
Cheng Pan
9486844610
HADOOP-16928. Make javadoc work on Java 17 (#6976) 
Contributed by Cheng Pan
 2024-09-04 11:50:59 +01:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)" 
This reverts commit fa9bb0d1ac.
 2024-08-29 14:42:03 +05:30
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953) 
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
 2024-08-15 16:44:54 +01:00
Tsz-Wo Nicholas Sze
b189ef8197
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6954) 2024-08-05 10:42:12 -07:00
Tsz-Wo Nicholas Sze
e48cd0e987 Revert "HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933)" 
This reverts commit 7638b4727e.
 2024-07-21 19:07:19 +08:00
Tsz-Wo Nicholas Sze
a5eb5e9611
HDFS-17576. Support user defined auth Callback. (#6945) 2024-07-20 15:21:06 +08:00
Tsz-Wo Nicholas Sze
7638b4727e
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933) 2024-07-20 15:16:01 +08:00
Heagan A
2fbbfe3cc9
HDFS-17546. Implementing HostsFileReader timeout (#6873) 2024-06-14 20:47:21 -07:00
fuchaohong
ddb805951e
HDFS-17471. Correct the percentage of sample range. (#6742). Contributed by fuchaohong. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-04-30 12:18:47 +08:00
Tsz-Wo Nicholas Sze
78987a71a6
HADOOP-19151. Support configurable SASL mechanism. (#6740) 2024-04-29 10:02:23 -07:00
huhaiyang
6ccb223c9c
HDFS-17461. Fix spotbugs in PeerCache#getInternal (#6721). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-04-12 19:53:25 +08:00
huhaiyang
81b05977f2
HDFS-17455. Fix Client throw IndexOutOfBoundsException in DFSInputStream#fetchBlockAt (#6710). Contributed by Haiyang Hu. 
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-04-11 18:04:57 +08:00
PJ Fanning
f7d1ec2d9e
HADOOP-19077. Remove use of javax.ws.rs.core.HttpHeaders (#6554). Contributed by PJ Fanning 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-04-01 12:43:39 +05:30
Alex
50370769cd
HDFS-17429. Fixing wrong log file name in datatransfer Sender.java (#6670) Contributed by Zhongkun Wu. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-03-27 06:59:15 +08:00
huhaiyang
77c600d769
HDFS-17426. Remove Invalid FileSystemECReadStats logic in DFSInputStream (#6628) 2024-03-21 10:33:41 +08:00
ritegarg
58afe43769
HDFS-17299. Adding rack failure tolerance when creating a new file (#6566) 2024-03-06 13:08:05 -08:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. (#6430) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-03-02 21:35:24 +09:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider (#6510) 
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
 2024-02-13 10:49:39 +09:00
hfutatzhanghb
4f4b846986
HDFS-17359. EC: recheck failed streamers should only after flushing all packets. (#6503). Contributed by farmmamba. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2024-02-01 23:53:37 +09:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. (#6368). Contributed by farmmamba. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-22 11:50:51 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException (#6446) 
Print a warn log message for read retries and only print the full stack trace for a read request failure.

Contributed by: Xing Lin
 2024-01-18 18:03:28 -08:00
Lei313
cc4c4be1b7
HDFS-17331:Fix Blocks are always -1 and DataNode version are always UNKNOWN in federationhealth.html (#6429). Contributed by lei w. 
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-18 21:10:54 +08:00
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. (#6357). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-13 20:34:02 +08:00
hfutatzhanghb
d5468d84ba
HDFS-17283. Change the name of variable SECOND in HdfsClientConfigKeys. (#6339). Contributed by farmmamba. 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
 2024-01-04 19:53:47 +08:00
hfutatzhanghb
415e9bdfbd
HDFS-17273. Change the way of computing some local variables duration for better debugging (#6321). Contributed by farmmamba. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-12-24 16:57:04 +08:00
hfutatzhanghb
562c42c86a
HDFS-17242. Make congestion backoff time configurable. (#6227) 
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-12-13 20:56:53 +08:00
PJ Fanning
f609460bda
HADOOP-18957. Use StandardCharsets.UTF_8 (#6231). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-11-20 23:44:48 +05:30
LiuGuH
342e6caba1
HDFS-17249. Fix TestDFSUtil.testIsValidName() unit test failure (#6249) 
Contributed by liuguanghua.
 2023-11-13 18:16:31 +00:00
Tom
f58945d7d1
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and implementations (#6198) 
The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
 2023-11-08 14:25:21 +00:00
Gautham B A
cbb153b69e
HDFS-17246. Fix shaded client for building Hadoop on Windows (#5943) 2023-11-01 09:10:15 -07:00
Hiroaki Segawa
93a3c6e2cd
HDFS-17024. Potential data race introduced by HDFS-15865 (#6223) 2023-10-26 22:25:00 -07:00
Steve Loughran
9bc159f4ac
HADOOP-18487. Make protobuf 2.5 an optional runtime dependency. (#4996) 
Protobuf 2.5 JAR is no longer needed at runtime. 

The option common.protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.

* New package org.apache.hadoop.ipc.internal for internal only protobuf classes
  ...with a ShadedProtobufHelper in there which has shaded protobuf refs
  only, so guaranteed not to need protobuf-2.5 on the CP
* All uses of org.apache.hadoop.ipc.ProtobufHelper have
  been replaced by uses of org.apache.hadoop.ipc.internal.ShadedProtobufHelper
* The scope of protobuf-2.5 is set by the option common.protobuf2.scope
  In this patch is it is still "compile"
* There is explicit reference to it in modules where it may be needed.
*  The maven scope of the dependency can be set with the common.protobuf2.scope
   option. It can be set to "provided" in a build:
       -Dcommon.protobuf2.scope=provided
* Add new ipc(callable) method to catch and convert shaded protobuf
  exceptions raised during invocation of the supplied lambda expression
* This is adopted in the code where the migration is not traumatically
  over-complex. RouterAdminProtocolTranslatorPB is left alone for this
  reason.

Contributed by Steve Loughran
 2023-10-13 13:48:38 +01:00
PJ Fanning
c16484ffb2
HADOOP-18890. Remove use of okhttp in runtime code (#6057) 
Contributed by PJ Fanning
 2023-09-19 12:38:36 +01:00
hfutatzhanghb
18f9989ff2
HDFS-17192. Add bock info when constructing remote block reader meets IOException. (#6081). Contributed by farmmamba. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-09-18 20:56:45 +08:00
zhtttylz
9857fbcd61
HDFS-17168. Support getTrashRoots API in WebHDFS (#5992) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-09-02 15:24:52 +08:00
zhangshuyan
10b1d7340b
HDFS-17154. EC: Fix bug in updateBlockForPipeline after failover. (#5941). Contributed by Shuyan Zhang. 
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-08-16 20:29:19 +08:00
Xing Lin
80fefd093f
HDFS-17067 Use BlockingThreadPoolExecutorService for nnProbingThreadPool in ObserverReadProxy (#5803) 2023-07-20 10:46:41 -07:00
zhtttylz
dfb351c3a8
HDFS-17083. Support getErasureCodeCodecs API in WebHDFS (#5836). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-07-14 11:53:36 +05:30
Xing Lin
ab47a658bd
HDFS-17030. Limit wait time for getHAServiceState in ObserverReaderProxy (#5700) 2023-06-14 10:53:56 -07:00
zhangshuyan
9c989515ba
HDFS-17037. Consider nonDfsUsed when running balancer. (#5715). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-06-09 16:19:08 +08:00
zhtttylz
d9980ab40f
HDFS-17029. Support getECPolices API in WebHDFS (#5698). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-06-05 17:33:37 +05:30
Ayush Saxena
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module (#5503)" 
This reverts commit 03a499821c.
 2023-06-05 09:34:40 +05:30
Viraj Jasani
03a499821c
HADOOP-18207. Introduce hadoop-logging module (#5503) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
 2023-06-02 18:07:34 -07:00
Simbarashe Dzinamarira
d92a5815f4
HDFS-17027. RBF: Adds auto-msync support for clients connecting to routers. (#5693) 2023-05-31 10:20:19 -07:00
Keyao Li
339bc7b3a6
HDFS-16653. Improve error messages in ShortCircuitCache. (#5568). Contributed by ECFuzz. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 07:43:18 +05:30
Xianming Lei
441fb23293
HDFS-17018. Improve dfsclient log format. (#5668). Contributed by Xianming Lei. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-19 06:25:43 +05:30