Commit Graph

27427 Commits

Author SHA1 Message Date
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006)
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
2024-09-24 16:50:22 +01:00
Ayush Saxena
f90a703e48
HADOOP-19165. Drop protobuf 2.5.0 from the distribution (#7051). Contributed by Ayush Saxena. 2024-09-24 20:58:41 +05:30
Peter Szucs
b078f86d69
YARN-11733. Fix the order of updating CPU controls with cgroup v1 (#7069) 2024-09-24 17:13:28 +02:00
Attila Magyar
68315744f0
HDFS-17040. Namenode web UI should set content type to application/octet-stream when uploading a file. (#5721) 2024-09-23 12:21:38 -07:00
Steve Loughran
37a74f0692
HADOOP-19285. [ABFS] Restore ETAGS_AVAILABLE to abfs path capabilities (#7064)
Caused by HADOOP-19131  

Contributed by: Steve Loughran
2024-09-23 12:52:05 -05:00
Felix Nguyen
fccc268cde
HADOOP-19283. Move all DistCp execution logic to execute() (#7060)
Co-authored-by: Felix Nguyen <kokonguyen191@gmail.com>
2024-09-23 15:39:56 +08:00
gavin.wang
f5345b881d
YARN-11560. Fix NPE bug when multi-node enabled with schedule asynchronously (#6021). Contributed by wangzhongwei. 2024-09-23 13:09:15 +05:30
cxzl25
ae63d8edd5
HDFS-17621. Make PathIsNotEmptyDirectoryException terse (#7036). Contributed by dzcxzl. 2024-09-23 13:07:30 +05:30
dannytbecker
50fcfd8b29
HDFS-17526. getMetadataInputStream should use getShareDeleteFileInputStream for windows (#6826). Contributed by Danny Becker 2024-09-23 12:16:25 +05:30
Arjun Mohnot
d8ca2dbe34
YARN-11730. Mark unreported nodes as LOST on RM Startup/HA failover (#7049) Contributed by Arjun Mohnot.
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-09-23 10:46:20 +08:00
huhaiyang
81faae6343
Revert "HDFS-17611. Move all DistCp execution logic to execute() (#7025)" (#7059)
This reverts commit ff75aa52f3.
2024-09-23 10:44:14 +08:00
Felix Nguyen
ff75aa52f3
HDFS-17611. Move all DistCp execution logic to execute() (#7025)
---------

Co-authored-by: Felix Nguyen <kokonguyen191@gmail.com>
2024-09-23 10:36:20 +08:00
Ayush Saxena
28538d628e
HADOOP-19164. Hadoop CLI MiniCluster is broken (#7050). Contributed by Ayush Saxena.
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
2024-09-21 21:26:51 +05:30
manika137
6bcc254123
HADOOP-19279. ABFS: Disabling Apache Http Client as Default Http Client for ABFS Driver(#7055)
As part of work done under HADOOP-19120 [ABFS]: ApacheHttpClient adaptation as network library - ASF JIRA
Apache HTTP Client was introduced as an alternative Network Library that can be used with ABFS Driver. Earlier JDK Http Client was the only supported network library.

Apache HTTP Client was found to be more helpful in terms of controls and knobs it provides to manage the Network aspects of the driver better. Hence, the default Network Client was made to be used with the ABFS Driver.

Recently while running scale workloads, we observed a regression where some unexpected wait time was observed while establishing connections. A possible fix has been identified and we are working on getting it fixed.
There was also a possible NPE scenario was identified on the new network client code.

Until we are done with the code fixes and revalidated the whole Apache client flow, we would like to make JDK Client as default client again. The support will still be there, but it will be disabled behind a config.

Contributed by: manika137
2024-09-20 16:38:56 -05:00
Steve Loughran
ee2e5ac4e4
HADOOP-19272. S3A: AWS SDK 2.25.53 warnings logged by transfer manager (#7048)
Disables all logging below error in the AWS SDK Transfer Manager.

This is done in ClientManagerImpl construction so is automatically done
during S3A FS initialization.

ITests verify that
* It is possible to restore the warning log. This verifies the validity of
  the test suite, and will identify when an SDK update fixes this regression.
* Constructing an S3A FS instance will disable the logging.

The log manipulation code is lifted from Cloudstore, where it was used to
dynamically enable logging. It uses reflection to load the Log4J binding;
all uses of the API catch and swallow exceptions.
This is needed to avoid failures when running against different log backends

This is an emergency fix -we could come up with a better design for
the reflection based code using the new DynMethods classes.
But this is based on working code, which is always good.

Contributed by Steve Loughran
2024-09-19 13:50:06 +01:00
Benjamin Teke
d1311e52f7
YARN-11709. NodeManager should be marked unhealthy on localizer config issues (#7043) 2024-09-17 14:24:11 +02:00
Doroszlai, Attila
182feb11a0
HADOOP-19277. Files and directories mixed up in TreeScanResults#dump (#7047) 2024-09-17 12:25:57 +02:00
Pranav Saxena
4d968add52
HADOOP-19271. NPE in AbfsManagedApacheHttpConnection.toString() when not connected (#7040)
Contributed by: Pranav Saxena
2024-09-16 12:21:20 -05:00
Steve Loughran
ea6e0f7cd5
HADOOP-19221. S3A: Unable to recover from failure of multipart block upload attempt (#6938)
This is a major change which handles 400 error responses when uploading
large files from memory heap/buffer (or staging committer) and the remote S3
store returns a 500 response from a upload of a block in a multipart upload.

The SDK's own streaming code seems unable to fully replay the upload;
at attempts to but then blocks and the S3 store returns a 400 response

    "Your socket connection to the server was not read from or written to
     within the timeout period. Idle connections will be closed.
     (Service: S3, Status Code: 400...)"

There is an option to control whether or not the S3A client itself
attempts to retry on a 50x error other than 503 throttling events
(which are independently processed as before)

Option:  fs.s3a.retry.http.5xx.errors
Default: true

500 errors are very rare from standard AWS S3, which has a five nines
SLA. It may be more common against S3 Express which has lower
guarantees.

Third party stores have unknown guarantees, and the exception may
indicate a bad server configuration. Consider setting
fs.s3a.retry.http.5xx.errors to false when working with
such stores.

Signification Code changes:

There is now a custom set of implementations of
software.amazon.awssdk.http.ContentStreamProvidercontent in
the class org.apache.hadoop.fs.s3a.impl.UploadContentProviders.

These:

* Restart on failures
* Do not copy buffers/byte buffers into new private byte arrays,
  so avoid exacerbating memory problems..

There new IOStatistics for specific http error codes -these are collected
even when all recovery is performed within the SDK.
  
S3ABlockOutputStream has major changes, including handling of
Thread.interrupt() on the main thread, which now triggers and briefly
awaits cancellation of any ongoing uploads.

If the writing thread is interrupted in close(), it is mapped to
an InterruptedIOException. Applications like Hive and Spark must
catch these after cancelling a worker thread.

Contributed by Steve Loughran
2024-09-13 20:02:14 +01:00
Smith Cruise
c835adb3a8
HADOOP-19201 S3A. Support external-id in assume role (#6876)
The option fs.s3a.assumed.role.external.id sets the external id for calls of AssumeRole to the STS service

Contributed by Smith Cruise
2024-09-10 15:38:32 +01:00
K0K0V0K
c9e9bce361
YARN-11729. Broken 'AM Node Web UI' link on App details page (#7030) 2024-09-09 16:33:40 +02:00
Saikat Roy
6881d12da4
HADOOP-19262: Upgrade wildfly-openssl:1.1.3.Final to 2.1.4.Final to support Java17+ (#7026)
Contributed by Saikat Roy
2024-09-09 15:14:03 +01:00
Benjamin Teke
8c41fbcaf5
Revert "YARN-11709. NodeManager should be shut down or blacklisted when it ca…" (#7028)
Some checks failed
website / build (push) Has been cancelled
This reverts commit f00094203b.
2024-09-07 08:48:38 +02:00
PJ Fanning
a00b1c06f3
HADOOP-19269. Upgrade maven-shade-plugin 3.6.0 (#7029)
Contributed by PJ Fanning
2024-09-05 20:29:44 +01:00
Steve Loughran
57e62ae07f
Revert "YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)"
This reverts commit 6c01490f14.
2024-09-05 14:35:50 +01:00
Shintaro Onuma
1f302e83fd
HADOOP-18938. S3A: Fix endpoint region parsing for vpc endpoints. (#6466)
Contributed by Shintaro Onuma
2024-09-05 14:14:04 +01:00
Syed Shameerur Rahman
6c01490f14
YARN-11664. Remove HDFS Binaries/Jars Dependency From Yarn (#6631)
To support YARN deployments in clusters without HDFS
some changes have been made in packaging

* new hadoop-common class org.apache.hadoop.fs.HdfsCommonConstants
* hdfs class org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair moved
  from hdfs-client to hadoop-common
* YARN handlers for DSQuotaExceededException replaced by use of superclass
  ClusterStorageCapacityExceededException.  

Contributed by Syed Shameerur Rahman
2024-09-04 13:26:42 +01:00
Cheng Pan
9486844610
HADOOP-16928. Make javadoc work on Java 17 (#6976)
Contributed by Cheng Pan
2024-09-04 11:50:59 +01:00
Steve Loughran
3bbfb2be08
HADOOP-19257. S3A: ITestAssumeRole.testAssumeRoleBadInnerAuth failure (#7021)
Remove the error string matched on so that no future message change
from AWS will trigger a regression

Contributed by Steve Loughran
2024-09-03 21:20:47 +01:00
zhengchenyu
1655acc5e2
HADOOP-19250. [Addendum] Fix test TestServiceInterruptHandling.testRegisterAndRaise. (#7008)
Contributed by Chenyu Zheng
2024-08-30 12:05:13 +01:00
Steve Loughran
b404c8c8f8
HADOOP-19252. Upgrade hadoop-thirdparty to 1.3.0 (#7007)
Update the version of hadoop-thirdparty to 1.3.0
across all shaded artifacts used.

This synchronizes the shaded protobuf library with those of
all other shaded artifacts (guava, avro)

Contributed by Steve Loughran
2024-08-30 11:50:51 +01:00
litao
a962aa37e0
HDFS-17599. EC: Fix the mismatch between locations and indices for mover (#6980) 2024-08-30 12:56:33 +08:00
Ayush Saxena
0837c84a9f
Revert "HADOOP-19231. Add JacksonUtil to manage Jackson classes (#6953)"
This reverts commit fa9bb0d1ac.
2024-08-29 14:42:03 +05:30
Cheng Pan
0aab1a2976
HADOOP-19248. Protobuf code generate and replace should happen together (#6975)
Contributed by Cheng Pan
2024-08-28 20:18:46 +01:00
K0K0V0K
e4ee3d560b
YARN-10345 HsWebServices containerlogs does not honor ACLs for completed jobs (#7013)
- following rest apis did not have access control
- - /ws/v1/history/containerlogs/{containerid}/{filename}
- - /ws/v1/history/containers/{containerid}/logs

Change-Id: I434f6138966ab22583d356509e40b70d328d9e7c
2024-08-27 17:55:07 +02:00
Sung Dong Kim
89e38f08ae
HDFS-17573. Allow turn on both FSImage parallelization and compression (#6929). Contributed by Sung Dong Kim.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-08-25 17:51:14 +08:00
Kevin Cai
5745a7dd75
HDFS-16084. Fix getJNIEnv crash due to incorrect state set to tls var (#6969). Contributed by Kevin Cai.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-08-25 16:47:05 +08:00
slfan1989
6be04633b5
YARN-11711. Clean Up ServiceScheduler Code. (#6977) Contributed by Shilun Fan.
Reviewed-by: 	Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-08-22 17:49:42 +08:00
Heagan A
f6c45e0bcf
HDFS-17546. Follow-up backport from branch3.3 (#6908)
HDFS-17546. Follow-up backport from branch3.3
2024-08-21 13:11:32 -07:00
Carl Levasseur
68fcd7234c
HADOOP-18542. Keep MSI tenant ID and client ID optional (#4262)
Contributed by Carl Levasseur
2024-08-21 14:15:28 +01:00
Tsz-Wo Nicholas Sze
012ae9d1aa
HDFS-17606. Do not require implementing CustomizedCallbackHandler. (#7005) 2024-08-20 16:32:53 -07:00
Anuj Modi
b15ed27cfb
HADOOP-19187: [ABFS][FNSOverBlob] AbfsClient Refactoring to Support Multiple Implementation of Clients. (#6879)
Refactor AbfsClient into DFS and Blob Client.

Contributed by Anuj Modi
2024-08-20 18:07:07 +01:00
dhavalshah9131
33c9ecb652
HADOOP-19249. KMSClientProvider raises NPE with unauthed user (#6984)
KMSClientProvider raises a NullPointerException when an unauthorised user
tries to perform the key operation

Contributed by Dhaval Shah
2024-08-20 14:03:05 +01:00
Steve Loughran
2fd7cf53fa
HADOOP-19253. Google GCS compilation fails due to VectorIO changes (#7002)
Fixes a compilation failure caused by HADOOP-19098

Restore original sortRanges() method signature,
  FileRange[] sortRanges(List<? extends FileRange>)

This ensures that google GCS connector will compile again.
It has also been marked as Stable so it is left alone

The version returning List<? extends FileRange>
has been renamed sortRangeList()

Contributed by Steve Loughran
2024-08-19 19:58:15 +01:00
Stephen O'Donnell
df08e0de41
HDFS-17605. Reduce memory overhead of TestBPOfferService (#6996) 2024-08-19 11:35:11 +01:00
zhengchenyu
e5b76dc99f
HADOOP-19180. EC: Fix calculation errors caused by special index order (#6813). Contributed by zhengchenyu.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-08-19 12:40:45 +08:00
PJ Fanning
59dba6e1bd
HADOOP-19134. Use StringBuilder instead of StringBuffer. (#6692). Contributed by PJ Fanning 2024-08-18 21:29:12 +05:30
slfan1989
b5f88990b7
HADOOP-19136. Upgrade commons-io to 2.16.1. (#6704)
Contributed by Shilun Fan.
2024-08-16 19:42:26 +01:00
zhengchenyu
bf804cb64b
HADOOP-19250. Fix test TestServiceInterruptHandling.testRegisterAndRaise (#6987)
Contributed by Chenyu Zheng
2024-08-16 17:16:28 +01:00
Ferenc Erdelyi
f00094203b
YARN-11709. NodeManager should be shut down or blacklisted when it cacannot run program /var/lib/yarn-ce/bin/container-executor (#6960) 2024-08-16 16:33:10 +02:00