Commit Graph

27376 Commits

Author SHA1 Message Date
PJ Fanning
fa9bb0d1ac
HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)
New class org.apache.hadoop.util.JacksonUtil centralizes construction of
Jackson ObjectMappers and JsonFactories.

Contributed by PJ Fanning
2024-08-15 16:44:54 +01:00
Steve Loughran
55a576906d
HADOOP-19131. Assist reflection IO with WrappedOperations class (#6686)
1. The class WrappedIO has been extended with more filesystem operations

- openFile()
- PathCapabilities
- StreamCapabilities
- ByteBufferPositionedReadable

All these static methods raise UncheckedIOExceptions rather than
checked ones.

2. The adjacent class org.apache.hadoop.io.wrappedio.WrappedStatistics
provides similar access to IOStatistics/IOStatisticsContext classes
and operations.

Allows callers to:
* Get a serializable IOStatisticsSnapshot from an IOStatisticsSource or
  IOStatistics instance
* Save an IOStatisticsSnapshot to file
* Convert an IOStatisticsSnapshot to JSON
* Given an object which may be an IOStatisticsSource, return an object
  whose toString() value is a dynamically generated, human readable summary.
  This is for logging.
* Separate getters to the different sections of IOStatistics.
* Mean values are returned as a Map.Pair<Long, Long> of (samples, sum)
  from which means may be calculated.

There are examples of the dynamic bindings to these classes in:

org.apache.hadoop.io.wrappedio.impl.DynamicWrappedIO
org.apache.hadoop.io.wrappedio.impl.DynamicWrappedStatistics

These use DynMethods and other classes in the package
org.apache.hadoop.util.dynamic which are based on the
Apache Parquet equivalents.
This makes re-implementing these in that library and others
which their own fork of the classes (example: Apache Iceberg)

3. The openFile() option "fs.option.openfile.read.policy" has
added specific file format policies for the core filetypes

* avro
* columnar
* csv
* hbase
* json
* orc
* parquet

S3A chooses the appropriate sequential/random policy as a 

A policy `parquet, columnar, vector, random, adaptive` will use the parquet policy for
any filesystem aware of it, falling back to the first entry in the list which
the specific version of the filesystem recognizes

4. New Path capability fs.capability.virtual.block.locations

Indicates that locations are generated client side
and don't refer to real hosts.

Contributed by Steve Loughran
2024-08-14 14:43:00 +01:00
Viraj Jasani
fa83c9a805
HADOOP-19072 S3A: Override fs.s3a.performance.flags for tests (ADDENDUM 2) (#6993)
Second followup to #6543; all hadoop-aws integration tests complete correctly even when 

fs.s3a.performance.flags = *

Contributed by Viraj Jasani
2024-08-14 10:57:44 +01:00
Viraj Jasani
74ff00705c
HADOOP-19072. S3A: Override fs.s3a.performance.flags for tests (ADDENDUM) (#6985)
This is a followup to #6543 which ensures all test pass in configurations where 
fs.s3a.performance.flags is set to "*" or contains "mkdirs"

Contributed by VJ Jasani
2024-08-12 14:16:44 +01:00
Viraj Jasani
321a6cc55e
HADOOP-19072. S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir (#6543)
If the flag list in fs.s3a.performance.flags includes "mkdir"
then the safety check of a walk up the tree to look for a parent directory,
-done to verify a directory isn't being created under a file- are skipped.

This saves the cost of multiple list operations.

Contributed by Viraj Jasani
2024-08-08 17:48:51 +01:00
Masatake Iwasaki
2a50911734
HADOOP-17609. Make SM4 support optional for OpenSSL native code. (#3019)
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2024-08-08 21:03:05 +09:00
Tsz-Wo Nicholas Sze
b189ef8197
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6954) 2024-08-05 10:42:12 -07:00
Cheng Pan
59d5e0bb2e
HADOOP-19244. Pullout arch-agnostic maven javadoc plugin configurations in hadoop-common (#6970) Contributed by Cheng Pan.
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-08-05 15:30:36 +08:00
zhengchenyu
b08d492abd
HADOOP-19246. Update the yasm rpm download address (#6973)
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2024-08-05 09:57:16 +08:00
Steve Loughran
2cf4d638af
HADOOP-19245. S3ABlockOutputStream no longer sends progress events in close() (#6974)
Contributed by Steve Loughran
2024-08-02 16:01:03 +01:00
PJ Fanning
c593c17255
HADOOP-19237. Upgrade to dnsjava 3.6.1 due to CVEs (#6961)
Contributed by P J Fanning
2024-08-01 20:07:36 +01:00
Takanobu Asanuma
059e996c02
HDFS-17591. RBF: Router should follow X-FRAME-OPTIONS protection setting (#6963) 2024-07-30 10:14:33 +09:00
Mukund Thakur
038636a1b5
HADOOP-19238. Fix create-release script for arm64 based MacOS (#6962)
Contributed by Mukund Thakur
2024-07-29 19:45:14 +01:00
Steve Loughran
a5806a9e7b
HADOOP-19161. S3A: option "fs.s3a.performance.flags" to take list of performance flags (#6789)
1. Configuration adds new method `getEnumSet()` to get a set of enum values from
   a configuration string.
   <E extends Enum<E>> EnumSet<E> getEnumSet(String key, Class<E> enumClass, boolean ignoreUnknown)

   Whitespace is ignored, case is ignored and the value "*" is mapped to "all values of the enum".
   If "ignoreUnknown" is true then when parsing, unknown values are ignored.
   This is recommended for forward compatiblity with later versions.

2. This support is implemented in org.apache.hadoop.fs.s3a.impl.ConfigurationHelper -it can be used
    elsewhere in the hadoop codebase.

3. A new private FlagSet class in hadoop common manages a set of enum flags.

     It implements StreamCapabilities and can be probed for a specific option being set
    (with a prefix)


S3A adds an option fs.s3a.performance.flags which builds a FlagSet with enum
type PerformanceFlagEnum

* which initially contains {Create, Delete, Mkdir, Open}
* the existing fs.s3a.create.performance option sets the flag "Create".
* tests which configure fs.s3a.create.performance MUST clear
  fs.s3a.performance.flags in test setup.

Future performance flags are planned, with different levels of safety
and/or backwards compatibility.

Contributed by Steve Loughran
2024-07-29 11:33:51 +01:00
Raphael Azzolini
4525c7e35e
HADOOP-19197. S3A: Support AWS KMS Encryption Context (#6874)
The new property fs.s3a.encryption.context allow users to specify the AWS KMS Encryption Context to be used in S3A.

The value of the encryption context is a key/value string that will be Base64 encoded and set in the parameter ssekmsEncryptionContext from the S3 client.

Contributed by Raphael Azzolini
2024-07-23 17:09:04 +01:00
Aswin M Prabhu
e2a0dca43b
HDFS-16690. Automatically format unformatted JNs with JournalNodeSyncer (#6925). Contributed by Aswin M Prabhu.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-23 20:55:57 +08:00
Viraj Jasani
e000cbf277
HADOOP-19218. Addendum. Update TestFSNamesystemLockReport to exclude hostname resolution from regex. (#6951). Contributed by Viraj Jasani.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-23 20:47:36 +08:00
Pranav Saxena
b60497ff41
HADOOP-19120. ApacheHttpClient adaptation in ABFS. (#6633)
Apache httpclient 4.5.x is the new default implementation of http connections;
this supports a large configurable pool of connections along with
the ability to limit their lifespan.

The networking library can be chosen using the configuration
option fs.azure.networking.library

The supported values are
- APACHE_HTTP_CLIENT : Use Apache HttpClient [Default]
- JDK_HTTP_URL_CONNECTION : Use JDK networking library

Important: unless the networking library is switched back to
the JDK, the apache httpcore and httpclient must be on the classpath

Contributed by Pranav Saxena
2024-07-22 19:03:51 +01:00
Tsz-Wo Nicholas Sze
e48cd0e987 Revert "HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933)"
This reverts commit 7638b4727e.
2024-07-21 19:07:19 +08:00
fuchaohong
1577f57d4c
HADOOP-19228. ShellCommandFencer#setConfAsEnvVars should also replace '-' with '_'. (#6936). Contributed by fuchaohong.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-20 16:13:33 +08:00
Tsz-Wo Nicholas Sze
a5eb5e9611
HDFS-17576. Support user defined auth Callback. (#6945) 2024-07-20 15:21:06 +08:00
Tsz-Wo Nicholas Sze
9dad697dbc
HADOOP-19227. ipc.Server accelerate token negotiation only for the default mechanism. (#6949) 2024-07-20 15:18:22 +08:00
Tsz-Wo Nicholas Sze
7638b4727e
HDFS-17575. SaslDataTransferClient should use SaslParticipant to create messages. (#6933) 2024-07-20 15:16:01 +08:00
K0K0V0K
9a9ad6cc98
YARN-11705. Turn off Node Manager working directories validation by default (#6948)
Change-Id: I011c13c79719be97c7ebc028804f1fdab5eb34c4
2024-07-18 16:55:40 +02:00
fuchaohong
ebbe9628d3
HDFS-17558. RBF: Make maxSizeOfFederatedStateToPropagate work on setResponseHeaderState. (#6902)
Co-authored-by: fuchaohong <fuchaohong@chinatelecom.cn>
2024-07-17 09:43:00 -07:00
Viraj Jasani
1360c7574a
HADOOP-19218 Avoid DNS lookup while creating IPC Connection object (#6916). Contributed by Viraj Jasani.
Signed-off-by: Rushabh Shah <shahrs87@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-16 21:08:41 +08:00
gavin.wang
5730656660
HDFS-17574. Make NNThroughputBenchmark support human-friendly units about blocksize. (#6931). Contributed by wangzhongwei.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-16 20:57:50 +08:00
Anuj Modi
51cb858cc8
HADOOP-19208: [ABFS] Fixing logic to determine HNS nature of account to avoid extra getAcl() calls (#6893) 2024-07-15 21:51:54 +05:30
Cheng Pan
4f0ee9d67d
HADOOP-19222. Switch yum repo baseurl due to CentOS 7 sunset (#6932) Contributed by Cheng Pan.
Reviewed-by: Gautham Banasandra <gaurava@apache.org>
Signed-off-by:  Shilun Fan <slfan1989@apache.org>
2024-07-12 06:16:44 +08:00
zhengchenyu
8913d379fd
HDFS-17566. Got wrong sorted block order when StorageType is considered. (#6934). Contributed by Chenyu Zheng.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-11 17:41:24 +08:00
Yuanbo Liu
f211af30be
HDFS-17534. RBF: Support leader follower mode for multiple subclusters (#6861). Contributed by Yuanbo Liu.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-09 14:01:58 +05:30
gavin.wang
783a852029
HDFS-17555. Fix NumberFormatException of NNThroughputBenchmark when configured dfs.blocksize. (#6894). Contributed by wangzhongwei
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-09 13:52:15 +05:30
HarshitGupta11
b1d96f6101
HADOOP-19195. S3A: Upgrade aws sdk v2 to 2.25.53 (#6900)
Contributed by Harshit Gupta
2024-07-08 10:18:53 +01:00
huhaiyang
8ca4627a0d
HDFS-17557. Fix bug for TestRedundancyMonitor#testChooseTargetWhenAllDataNodesStop (#6897). Contributed by Haiyang Hu.
Some checks failed
website / build (push) Has been cancelled
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-06 13:18:12 +05:30
huhaiyang
5a8f70a72e
HDFS-17559. Fix the uuid as null in NameNodeMXBean (#6906). Contributed by Haiyang Hu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-06 13:16:25 +05:30
Cheng Pan
25e28b41cc
HADOOP-19216. Upgrade Guice from 4.0 to 5.1.0 to support Java 17 (#6913). Contributed by Cheng Pan.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-06 13:13:49 +05:30
Steve Loughran
4c55adbb6b
HADOOP-19205. S3A: initialization/close slower than with v1 SDK (#6892)
Adds new ClientManager interface/implementation which provides on-demand
creation of synchronous and asynchronous s3 clients, s3 transfer manager,
and in close() terminates these.

S3A FS is modified to
* Create a ClientManagerImpl instance and pass down to its S3Store.
* Use the same ClientManager interface against S3Store to demand-create
  the services.
* Only create the async client as part of the transfer manager creation,
  which will take place during the first rename() operation.
* Statistics on client creation count and duration are recorded.
+ Statistics on the time to initialize and shutdown the S3A FS are collected
  in IOStatistics for reporting.

Adds to hadoop common class
  LazyAtomicReference<T> implements CallableRaisingIOE<T>, Supplier<T>
and subclass
  LazyAutoCloseableReference<T extends AutoCloseable>
    extends LazyAtomicReference<T> implements AutoCloseable

These evaluate the Supplier<T>/CallableRaisingIOE<T> they were
constructed with on the first (successful) read of the the value.
Any exception raised during this operation will be rethrown, and on future
evaluations the same operation retried.

These classes implement the Supplier and CallableRaisingIOE
interfaces so can actually be used for to implement lazy function evaluation
as Haskell and some other functional languages do.

LazyAutoCloseableReference is AutoCloseable; its close() method will
close the inner reference if it is set

This class is used in ClientManagerImpl for the lazy S3 Cliehnt creation
and closure.

Contributed by Steve Loughran.
2024-07-05 16:38:37 +01:00
huhaiyang
ae76e9475c
HDFS-17564. EC: Fix the issue of inaccurate metrics when decommission mark busy DN. (#6911). Contributed by Haiyang Hu.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-07-05 20:45:01 +08:00
hfutatzhanghb
a57105462b
HADOOP-19215. Fix unit tests testSlowConnection and testBadSetup failed in TestRPC. (#6912). Contributed by farmmamba.
Reviewed-by: huhaiyang <huhaiyang926@126.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-07-05 12:11:39 +05:30
Steve Loughran
c33d868606
HADOOP-19210. S3A: Speed up some slow unit tests (#6907)
Speed up slow tests
* TestS3AAWSCredentialsProvider: decrease thread pool shutdown time
* TestS3AInputStreamRetry: reduce retry limit and intervals

Contributed by Steve Loughran
2024-07-02 11:34:45 +01:00
K0K0V0K
134dcf166f
YARN-11703. Validate accessibility of Node Manager working directories (#6903) 2024-06-27 16:21:28 +02:00
Yu Zhang
b4ddb2d3bb
HDFS-13603: do not propagate ExecutionException and add maxRetries limit to NameNode edek cache warmup (#6774) 2024-06-24 09:34:52 -07:00
HarshitGupta11
d3b98cb1b2
HADOOP-19194:Add test to find unshaded dependencies in the aws sdk (#6865)
The new test TestAWSV2SDK scans the aws sdk bundle.jar and prints out all classes
which are unshaded, so at risk of creating classpath problems

It does not fail the test if this holds, because the current SDKs
do ship with unshaded classes; the test would always fail.

The SDK upgrade process should include inspecting the output
of this test to see if it has got worse (do a before/after check).

Once the AWS SDK does shade everything, we can have this
test fail on any regression

Contributed by Harshit Gupta
2024-06-24 10:41:11 +01:00
Steve Loughran
8ac9c1839a
HADOOP-19203. WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions (#6885)
* WrappedIO methods raise UncheckedIOExceptions
*New class org.apache.hadoop.util.functional.FunctionalIO
 with wrap/unwrap and the ability to generate a
 java.util.function.Supplier around a CallableRaisingIOE.

Contributed by Steve Loughran
2024-06-19 18:47:29 +01:00
Hexiaoqiao
6545b7eeef
HDFS-17098. DatanodeManager does not handle null storage type properly. (#6840). Contributed by ConfX.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-06-19 20:58:57 +08:00
Steve Loughran
56c8aa5f1c
HADOOP-19204. VectorIO regression: empty ranges are now rejected (#6887)
- restore old outcome: no-op
- test this
- update spec

This is a critical fix for vector IO and MUST be cherrypicked to all branches with
that feature

Contributed by Steve Loughran
2024-06-19 12:05:24 +01:00
Tsz-Wo Nicholas Sze
1e6411c9ec
HDFS-17528. FsImageValidation: set txid when saving a new image (#6828) 2024-06-19 11:38:17 +08:00
slfan1989
9710a8d52f
YARN-11701. [Federation] Enhance Federation Cache Clean Conditions. (#6889) Contributed by Shilun Fan.
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2024-06-19 08:34:19 +08:00
Fateh Singh
90024d8cb1
HDFS-17439. Support -nonSuperUser for NNThroughputBenchmark: useful for testing auth frameworks such as Ranger (#6677) 2024-06-18 13:52:24 +01:00
Heagan A
2fbbfe3cc9
HDFS-17546. Implementing HostsFileReader timeout (#6873) 2024-06-14 20:47:21 -07:00