big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,471 Commits 2 Branches 2 Tags 264 MiB
7999db55da
Commit Graph

10185 Commits

Author SHA1 Message Date
Joseph Dell'Aringa
9657276492
HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (#7127) 
HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (#7127)

---------

Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
 2024-11-07 10:15:51 -08:00
muskan1012
f7651e2f63
HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 
Mockito is now at a JDK-17 compatible version.

Contributed by Muskan Mishra
 2024-11-05 17:35:53 +00:00
Lei313
e4789a2fd3
HDFS-17607. Reduce the number of times conf is loaded when DataNode startUp (#7012). Contributed by lei w. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-04 20:10:08 +08:00
hfutatzhanghb
4f3abd2f48
HDFS-17654. Fix bugs in TestRouterMountTable (#7137). Contributed by farmmamba. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-02 17:27:06 +08:00
Zhaobo Huang
00cddf5bea
HDFS-17646. Add Option to limit Balancer overUtilized nodes num in each iteration. (#7120). Contributed by Zhaobo Huang. 
Reviewed-by: Haiyang Hu <huhaiyang926@126.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-02 17:06:18 +08:00
Lei Yang
eb1e30395b
HDFS-17644:Add log when a node selection is rejected by BPP UpgradeDomain (#7109)
Some checks failed
website / build (push) Has been cancelled
Details
2024-10-26 17:29:43 +08:00
Gautham B A
d1ce965645
HDFS-17636. Don't add declspec for Windows (#7096) 
* Windows doesn't want the
  macro _JNI_IMPORT_OR_EXPORT_
  to be defined in the function
  definition. It fails to compile with
  the following error -
  "definition of dllimport function
  not allowed".
* However, Linux needs it. Hence,
  we're going to add this macro
  based on the OS.
* Also, we'll be compiling the `hdfs`
  target as an object library so that
  we can avoid linking to `jvm`
  library for `get_jni_test` target.
 2024-10-22 23:15:23 +05:30
Felix Nguyen
09b348753f
HDFS-17634. RBF: Fix web UI missing DN last block report (#7080) 2024-10-22 19:54:16 +08:00
LiuGuH
6589d9f6aa
HDFS-17631. Fix RedundantEditLogInputStream.nextOp() state error when EditLogInputStream.skipUntil() throw IOException (#7066). Contributed by liuguanghua. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-10-16 21:16:18 +08:00
zhtttylz
1f0d9df887
HDFS-17637. Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-06 09:16:00 +08:00
litao
a9b7913d56
HDFS-17626. Reduce lock contention at datanode startup (#7053). Contributed by Tao Li. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-29 10:58:13 +08:00
fuchaohong
3d81dde28b
HDFS-17624. Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope. (#7042). Contributed by fuchaohong. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-27 16:56:09 +08:00
Sadanand Shenoy
49a495803a
HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551) 
Contributed by Sadanand Shenoy
 2024-09-25 17:54:09 +01:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
 2024-09-24 16:50:22 +01:00
Attila Magyar
68315744f0
HDFS-17040. Namenode web UI should set content type to application/octet-stream when uploading a file. (#5721) 2024-09-23 12:21:38 -07:00
cxzl25
ae63d8edd5
HDFS-17621. Make PathIsNotEmptyDirectoryException terse (#7036). Contributed by dzcxzl. 2024-09-23 13:07:30 +05:30
dannytbecker
50fcfd8b29
HDFS-17526. getMetadataInputStream should use getShareDeleteFileInputStream for windows (#6826). Contributed by Danny Becker 2024-09-23 12:16:25 +05:30
Ayush Saxena
28538d628e
HADOOP-19164. Hadoop CLI MiniCluster is broken (#7050). Contributed by Ayush Saxena. 
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
 2024-09-21 21:26:51 +05:30
Steve Loughran
57e62ae07f
Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)" 
This reverts commit 6c01490f14.
 2024-09-05 14:35:50 +01:00
Syed Shameerur Rahman
6c01490f14
YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631) 
To support YARN deployments in clusters without HDFS
some changes have been made in packaging

* new hadoop-common class org.apache.hadoop.fs.HdfsCommonConstants
* hdfs class org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair moved
  from hdfs-client to hadoop-common
* YARN handlers for DSQuotaExceededException replaced by use of superclass
  ClusterStorageCapacityExceededException.  

Contributed by Syed Shameerur Rahman
 2024-09-04 13:26:42 +01:00
Cheng Pan
9486844610
HADOOP-16928. Make javadoc work on Java 17 (#6976) 
Contributed by Cheng Pan
 2024-09-04 11:50:59 +01:00
litao
a962aa37e0
HDFS-17599. EC: Fix the mismatch between locations and indices for mover (#6980) 2024-08-30 12:56:33 +08:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)" 
This reverts commit fa9bb0d1ac.
 2024-08-29 14:42:03 +05:30
Sung Dong Kim
89e38f08ae
HDFS-17573. Allow turn on both FSImage parallelization and compression (#6929). Contributed by Sung Dong Kim. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-08-25 17:51:14 +08:00
Kevin Cai
5745a7dd75
HDFS-16084. Fix getJNIEnv crash due to incorrect state set to tls var (#6969). Contributed by Kevin Cai. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-08-25 16:47:05 +08:00
Heagan A
f6c45e0bcf
HDFS-17546. Follow-up backport from branch3.3 (#6908) 
HDFS-17546. Follow-up backport from branch3.3
 2024-08-21 13:11:32 -07:00
Tsz-Wo Nicholas Sze
012ae9d1aa
HDFS-17606. Do not require implementing CustomizedCallbackHandler. (#7005) 2024-08-20 16:32:53 -07:00
Stephen O'Donnell
df08e0de41
HDFS-17605. Reduce memory overhead of TestBPOfferService (#6996) 2024-08-19 11:35:11 +01:00
PJ Fanning
59dba6e1bd
HADOOP-19134. Use StringBuilder instead of StringBuffer. (#6692). Contributed by PJ Fanning 2024-08-18 21:29:12 +05:30
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953) 
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
 2024-08-15 16:44:54 +01:00
Steve Loughran
55a576906d
HADOOP-19131. Assist reflection IO with WrappedOperations class (#6686) 
1. The class WrappedIO has been extended with more filesystem operations

- openFile()
- PathCapabilities
- StreamCapabilities
- ByteBufferPositionedReadable

All these static methods raise UncheckedIOExceptions rather than
checked ones.

2. The adjacent class org.apache.hadoop.io.wrappedio.WrappedStatistics
provides similar access to IOStatistics/IOStatisticsContext classes
and operations.

Allows callers to:
* Get a serializable IOStatisticsSnapshot from an IOStatisticsSource or
  IOStatistics instance
* Save an IOStatisticsSnapshot to file
* Convert an IOStatisticsSnapshot to JSON
* Given an object which may be an IOStatisticsSource, return an object
  whose toString() value is a dynamically generated, human readable summary.
  This is for logging.
* Separate getters to the different sections of IOStatistics.
* Mean values are returned as a Map.Pair<Long, Long> of (samples, sum)
  from which means may be calculated.

There are examples of the dynamic bindings to these classes in:

org.apache.hadoop.io.wrappedio.impl.DynamicWrappedIO
org.apache.hadoop.io.wrappedio.impl.DynamicWrappedStatistics

These use DynMethods and other classes in the package
org.apache.hadoop.util.dynamic which are based on the
Apache Parquet equivalents.
This makes re-implementing these in that library and others
which their own fork of the classes (example: Apache Iceberg)

3. The openFile() option "fs.option.openfile.read.policy" has
added specific file format policies for the core filetypes

* avro
* columnar
* csv
* hbase
* json
* orc
* parquet

S3A chooses the appropriate sequential/random policy as a 

A policy `parquet, columnar, vector, random, adaptive` will use the parquet policy for
any filesystem aware of it, falling back to the first entry in the list which
the specific version of the filesystem recognizes

4. New Path capability fs.capability.virtual.block.locations

Indicates that locations are generated client side
and don't refer to real hosts.

Contributed by Steve Loughran
 2024-08-14 14:43:00 +01:00
Tsz-Wo Nicholas Sze
b189ef8197
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6954) 2024-08-05 10:42:12 -07:00
Takanobu Asanuma
059e996c02
HDFS-17591. RBF: Router should follow X-FRAME-OPTIONS protection setting (#6963) 2024-07-30 10:14:33 +09:00
Aswin M Prabhu
e2a0dca43b
HDFS-16690. Automatically format unformatted JNs with JournalNodeSyncer (#6925). Contributed by Aswin M Prabhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-23 20:55:57 +08:00
Viraj Jasani
e000cbf277
HADOOP-19218. Addendum. Update TestFSNamesystemLockReport to exclude hostname resolution from regex. (#6951). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-23 20:47:36 +08:00
Tsz-Wo Nicholas Sze
e48cd0e987 Revert "HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933)" 
This reverts commit 7638b4727e.
 2024-07-21 19:07:19 +08:00
Tsz-Wo Nicholas Sze
a5eb5e9611
HDFS-17576. Support user defined auth Callback. (#6945) 2024-07-20 15:21:06 +08:00
Tsz-Wo Nicholas Sze
7638b4727e
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933) 2024-07-20 15:16:01 +08:00
fuchaohong
ebbe9628d3
HDFS-17558. RBF: Make maxSizeOfFederatedStateToPropagate work on setResponseHeaderState. (#6902) 
Co-authored-by: fuchaohong <fuchaohong@chinatelecom.cn>
 2024-07-17 09:43:00 -07:00
gavin.wang
5730656660
HDFS-17574. Make NNThroughputBenchmark support human-friendly units about blocksize. (#6931). Contributed by wangzhongwei. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-16 20:57:50 +08:00
zhengchenyu
8913d379fd
HDFS-17566. Got wrong sorted block order when StorageType is considered. (#6934). Contributed by Chenyu Zheng. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-11 17:41:24 +08:00
Yuanbo Liu
f211af30be
HDFS-17534. RBF: Support leader follower mode for multiple subclusters (#6861). Contributed by Yuanbo Liu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-09 14:01:58 +05:30
gavin.wang
783a852029
HDFS-17555. Fix NumberFormatException of NNThroughputBenchmark when configured dfs.blocksize. (#6894). Contributed by wangzhongwei 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-09 13:52:15 +05:30
huhaiyang
8ca4627a0d
HDFS-17557. Fix bug for TestRedundancyMonitor#testChooseTargetWhenAllDataNodesStop (#6897). Contributed by Haiyang Hu.
Some checks failed
website / build (push) Has been cancelled
Details 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-06 13:18:12 +05:30
huhaiyang
5a8f70a72e
HDFS-17559. Fix the uuid as null in NameNodeMXBean (#6906). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-06 13:16:25 +05:30
huhaiyang
ae76e9475c
HDFS-17564. EC: Fix the issue of inaccurate metrics when decommission mark busy DN. (#6911). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-05 20:45:01 +08:00
Yu Zhang
b4ddb2d3bb
HDFS-13603: do not propagate ExecutionException and add maxRetries limit to NameNode edek cache warmup (#6774) 2024-06-24 09:34:52 -07:00
Hexiaoqiao
6545b7eeef
HDFS-17098. DatanodeManager does not handle null storage type properly. (#6840). Contributed by ConfX. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-06-19 20:58:57 +08:00
Tsz-Wo Nicholas Sze
1e6411c9ec
HDFS-17528. FsImageValidation: set txid when saving a new image (#6828) 2024-06-19 11:38:17 +08:00
Fateh Singh
90024d8cb1
HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful for testing auth frameworks such as Ranger (#6677) 2024-06-18 13:52:24 +01:00