big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,454 Commits 2 Branches 2 Tags 264 MiB
09b348753f
Commit Graph

10178 Commits

Author SHA1 Message Date
Felix Nguyen
09b348753f
HDFS-17634. RBF: Fix web UI missing DN last block report (#7080) 2024-10-22 19:54:16 +08:00
LiuGuH
6589d9f6aa
HDFS-17631. Fix RedundantEditLogInputStream.nextOp() state error when EditLogInputStream.skipUntil() throw IOException (#7066). Contributed by liuguanghua. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-10-16 21:16:18 +08:00
zhtttylz
1f0d9df887
HDFS-17637. Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-06 09:16:00 +08:00
litao
a9b7913d56
HDFS-17626. Reduce lock contention at datanode startup (#7053). Contributed by Tao Li. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-29 10:58:13 +08:00
fuchaohong
3d81dde28b
HDFS-17624. Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope. (#7042). Contributed by fuchaohong. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-27 16:56:09 +08:00
Sadanand Shenoy
49a495803a
HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551) 
Contributed by Sadanand Shenoy
 2024-09-25 17:54:09 +01:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
 2024-09-24 16:50:22 +01:00
Attila Magyar
68315744f0
HDFS-17040. Namenode web UI should set content type to application/octet-stream when uploading a file. (#5721) 2024-09-23 12:21:38 -07:00
cxzl25
ae63d8edd5
HDFS-17621. Make PathIsNotEmptyDirectoryException terse (#7036). Contributed by dzcxzl. 2024-09-23 13:07:30 +05:30
dannytbecker
50fcfd8b29
HDFS-17526. getMetadataInputStream should use getShareDeleteFileInputStream for windows (#6826). Contributed by Danny Becker 2024-09-23 12:16:25 +05:30
Ayush Saxena
28538d628e
HADOOP-19164. Hadoop CLI MiniCluster is broken (#7050). Contributed by Ayush Saxena. 
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
 2024-09-21 21:26:51 +05:30
Steve Loughran
57e62ae07f
Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)" 
This reverts commit 6c01490f14.
 2024-09-05 14:35:50 +01:00
Syed Shameerur Rahman
6c01490f14
YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631) 
To support YARN deployments in clusters without HDFS
some changes have been made in packaging

* new hadoop-common class org.apache.hadoop.fs.HdfsCommonConstants
* hdfs class org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair moved
  from hdfs-client to hadoop-common
* YARN handlers for DSQuotaExceededException replaced by use of superclass
  ClusterStorageCapacityExceededException.  

Contributed by Syed Shameerur Rahman
 2024-09-04 13:26:42 +01:00
Cheng Pan
9486844610
HADOOP-16928. Make javadoc work on Java 17 (#6976) 
Contributed by Cheng Pan
 2024-09-04 11:50:59 +01:00
litao
a962aa37e0
HDFS-17599. EC: Fix the mismatch between locations and indices for mover (#6980) 2024-08-30 12:56:33 +08:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)" 
This reverts commit fa9bb0d1ac.
 2024-08-29 14:42:03 +05:30
Sung Dong Kim
89e38f08ae
HDFS-17573. Allow turn on both FSImage parallelization and compression (#6929). Contributed by Sung Dong Kim. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-08-25 17:51:14 +08:00
Kevin Cai
5745a7dd75
HDFS-16084. Fix getJNIEnv crash due to incorrect state set to tls var (#6969). Contributed by Kevin Cai. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-08-25 16:47:05 +08:00
Heagan A
f6c45e0bcf
HDFS-17546. Follow-up backport from branch3.3 (#6908) 
HDFS-17546. Follow-up backport from branch3.3
 2024-08-21 13:11:32 -07:00
Tsz-Wo Nicholas Sze
012ae9d1aa
HDFS-17606. Do not require implementing CustomizedCallbackHandler. (#7005) 2024-08-20 16:32:53 -07:00
Stephen O'Donnell
df08e0de41
HDFS-17605. Reduce memory overhead of TestBPOfferService (#6996) 2024-08-19 11:35:11 +01:00
PJ Fanning
59dba6e1bd
HADOOP-19134. Use StringBuilder instead of StringBuffer. (#6692). Contributed by PJ Fanning 2024-08-18 21:29:12 +05:30
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953) 
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
 2024-08-15 16:44:54 +01:00
Steve Loughran
55a576906d
HADOOP-19131. Assist reflection IO with WrappedOperations class (#6686) 
1. The class WrappedIO has been extended with more filesystem operations

- openFile()
- PathCapabilities
- StreamCapabilities
- ByteBufferPositionedReadable

All these static methods raise UncheckedIOExceptions rather than
checked ones.

2. The adjacent class org.apache.hadoop.io.wrappedio.WrappedStatistics
provides similar access to IOStatistics/IOStatisticsContext classes
and operations.

Allows callers to:
* Get a serializable IOStatisticsSnapshot from an IOStatisticsSource or
  IOStatistics instance
* Save an IOStatisticsSnapshot to file
* Convert an IOStatisticsSnapshot to JSON
* Given an object which may be an IOStatisticsSource, return an object
  whose toString() value is a dynamically generated, human readable summary.
  This is for logging.
* Separate getters to the different sections of IOStatistics.
* Mean values are returned as a Map.Pair<Long, Long> of (samples, sum)
  from which means may be calculated.

There are examples of the dynamic bindings to these classes in:

org.apache.hadoop.io.wrappedio.impl.DynamicWrappedIO
org.apache.hadoop.io.wrappedio.impl.DynamicWrappedStatistics

These use DynMethods and other classes in the package
org.apache.hadoop.util.dynamic which are based on the
Apache Parquet equivalents.
This makes re-implementing these in that library and others
which their own fork of the classes (example: Apache Iceberg)

3. The openFile() option "fs.option.openfile.read.policy" has
added specific file format policies for the core filetypes

* avro
* columnar
* csv
* hbase
* json
* orc
* parquet

S3A chooses the appropriate sequential/random policy as a 

A policy `parquet, columnar, vector, random, adaptive` will use the parquet policy for
any filesystem aware of it, falling back to the first entry in the list which
the specific version of the filesystem recognizes

4. New Path capability fs.capability.virtual.block.locations

Indicates that locations are generated client side
and don't refer to real hosts.

Contributed by Steve Loughran
 2024-08-14 14:43:00 +01:00
Tsz-Wo Nicholas Sze
b189ef8197
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6954) 2024-08-05 10:42:12 -07:00
Takanobu Asanuma
059e996c02
HDFS-17591. RBF: Router should follow X-FRAME-OPTIONS protection setting (#6963) 2024-07-30 10:14:33 +09:00
Aswin M Prabhu
e2a0dca43b
HDFS-16690. Automatically format unformatted JNs with JournalNodeSyncer (#6925). Contributed by Aswin M Prabhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-23 20:55:57 +08:00
Viraj Jasani
e000cbf277
HADOOP-19218. Addendum. Update TestFSNamesystemLockReport to exclude hostname resolution from regex. (#6951). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-23 20:47:36 +08:00
Tsz-Wo Nicholas Sze
e48cd0e987 Revert "HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933)" 
This reverts commit 7638b4727e.
 2024-07-21 19:07:19 +08:00
Tsz-Wo Nicholas Sze
a5eb5e9611
HDFS-17576. Support user defined auth Callback. (#6945) 2024-07-20 15:21:06 +08:00
Tsz-Wo Nicholas Sze
7638b4727e
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933) 2024-07-20 15:16:01 +08:00
fuchaohong
ebbe9628d3
HDFS-17558. RBF: Make maxSizeOfFederatedStateToPropagate work on setResponseHeaderState. (#6902) 
Co-authored-by: fuchaohong <fuchaohong@chinatelecom.cn>
 2024-07-17 09:43:00 -07:00
gavin.wang
5730656660
HDFS-17574. Make NNThroughputBenchmark support human-friendly units about blocksize. (#6931). Contributed by wangzhongwei. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-16 20:57:50 +08:00
zhengchenyu
8913d379fd
HDFS-17566. Got wrong sorted block order when StorageType is considered. (#6934). Contributed by Chenyu Zheng. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-11 17:41:24 +08:00
Yuanbo Liu
f211af30be
HDFS-17534. RBF: Support leader follower mode for multiple subclusters (#6861). Contributed by Yuanbo Liu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-09 14:01:58 +05:30
gavin.wang
783a852029
HDFS-17555. Fix NumberFormatException of NNThroughputBenchmark when configured dfs.blocksize. (#6894). Contributed by wangzhongwei 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-09 13:52:15 +05:30
huhaiyang
8ca4627a0d
HDFS-17557. Fix bug for TestRedundancyMonitor#testChooseTargetWhenAllDataNodesStop (#6897). Contributed by Haiyang Hu.
Some checks failed
website / build (push) Has been cancelled
Details 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-06 13:18:12 +05:30
huhaiyang
5a8f70a72e
HDFS-17559. Fix the uuid as null in NameNodeMXBean (#6906). Contributed by Haiyang Hu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2024-07-06 13:16:25 +05:30
huhaiyang
ae76e9475c
HDFS-17564. EC: Fix the issue of inaccurate metrics when decommission mark busy DN. (#6911). Contributed by Haiyang Hu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-07-05 20:45:01 +08:00
Yu Zhang
b4ddb2d3bb
HDFS-13603: do not propagate ExecutionException and add maxRetries limit to NameNode edek cache warmup (#6774) 2024-06-24 09:34:52 -07:00
Hexiaoqiao
6545b7eeef
HDFS-17098. DatanodeManager does not handle null storage type properly. (#6840). Contributed by ConfX. 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-06-19 20:58:57 +08:00
Tsz-Wo Nicholas Sze
1e6411c9ec
HDFS-17528. FsImageValidation: set txid when saving a new image (#6828) 2024-06-19 11:38:17 +08:00
Fateh Singh
90024d8cb1
HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful for testing auth frameworks such as Ranger (#6677) 2024-06-18 13:52:24 +01:00
Heagan A
2fbbfe3cc9
HDFS-17546. Implementing HostsFileReader timeout (#6873) 2024-06-14 20:47:21 -07:00
hfutatzhanghb
4b1b16a846
HDFS-17551. Fix unit test failure caused by HDFS-17464. (#6883). Contributed by farmmamba. 2024-06-12 22:21:15 +05:30
Felix Nguyen
776c0a3ab9
HDFS-17539. Make TestFileChecksum fields static (#6853) 2024-06-11 15:26:21 +08:00
刘斌
6c08e8e2aa
HADOOP-19156. ZooKeeper based state stores use different ZK address configs. (#6767). Contributed by liu bin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-05-29 20:44:36 +08:00
Felix Nguyen
74d30a5dce
HDFS-17532. RBF: Allow router state store cache update to overwrite and delete in parallel (#6839) 2024-05-28 11:17:08 +08:00
Felix Nguyen
f5c5d35eb0
HDFS-17529. RBF: Improve router state store cache entry deletion (#6833) 2024-05-24 09:41:08 +08:00
hfutatzhanghb
fb156e8f05
HDFS-17464. Improve some logs output in class FsDatasetImpl (#6724) 2024-05-21 09:46:21 +08:00