Add support for S3 client side encryption (CSE).
CSE can configured in two modes:
- CSE-KMS where keys are provided by AWS KMS
- CSE-CUSTOM where custom keys are provided by implementing
a custom keyring.
CSE requires an encryption library:
amazon-s3-encryption-client-java.jar
This is _not_ included in the shaded bundle.jar
and is released separately.
The version used is currently 3.1.1
Contributed by Syed Shameerur Rahman.
* All field access is now via setter/getter methods
* To use Avro to marshal Serializable objects,
the packages they are in must be declared in the system property
"org.apache.avro.SERIALIZABLE_PACKAGES"
This is required to address
- CVE-2024-47561
- CVE-2023-39410
This change is not backwards compatible.
Contributed by Dominik Diedrich
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Attila Doroszlai <adoroszlai@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
Reviewed-by: Min Yan <yaommen@gmail.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
This moves Hadoop to Apache commons-collections4.
Apache commons-collections has been removed and is completely banned from the source code.
Contributed by Nihal Jain
Update the version of hadoop-thirdparty to 1.3.0
across all shaded artifacts used.
This synchronizes the shaded protobuf library with those of
all other shaded artifacts (guava, avro)
Contributed by Steve Loughran
- Critical: remove the obsolete exclusion list from hadoop-common.
- Diligence: expand the hadoop-project exclusion list to exclude
all ch.qos.logback artifacts
Contributed by Steve Loughran
Addresses
* CVE-2024-29857 - Importing an EC certificate with specially crafted F2m parameters can cause high CPU usage during parameter evaluation.
* CVE-2024-30171 - Possible timing based leakage in RSA based handshakes due to exception processing eliminated.
* CVE-2024-30172 - Crafted signature and public key can be used to trigger an infinite loop in the Ed25519 verification code.
* CVE-2024-301XX - When endpoint identification is enabled and an SSL socket is not created with an explicit hostname (as happens with HttpsURLConnection), hostname verification could be performed against a DNS-resolved IP address.
Contributed by PJ Fanning
This addresses two CVEs triggered by malformed archives
Important: Denial of Service CVE-2024-25710
Moderate: Denial of Service CVE-2024-26308
Contributed by PJ Fanning
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging
This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.
Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.
Contributed by Steve Loughran
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607).
Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
Cut out S3 Select
* leave public/unstable constants alone
* s3guard tool will fail with error
* s3afs. path capability will fail
* openFile() will fail with specific error
* s3 select doc updated
* Cut eventstream jar
* New test: ITestSelectUnsupported verifies new failure
handling above
Contributed by Steve Loughran
