yanmin
df979e70de
HADOOP-19297. [JDK17] Upgrade maven.plugin-tools.version to 3.10.2 ( #7125 ) Contributed by Min Yan.
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-11-05 07:46:35 +08:00
Christos Bisias
66baf1eb51
HADOOP-18682. Move hadoop docker scripts under the main source code ( #6483 ). Contributed by Christos Bisias.
2024-11-04 22:22:37 +05:30
Lei313
e4789a2fd3
HDFS-17607. Reduce the number of times conf is loaded when DataNode startUp ( #7012 ). Contributed by lei w.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-11-04 20:10:08 +08:00
hfutatzhanghb
4f3abd2f48
HDFS-17654. Fix bugs in TestRouterMountTable ( #7137 ). Contributed by farmmamba.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-11-02 17:27:06 +08:00
Zhaobo Huang
00cddf5bea
HDFS-17646. Add Option to limit Balancer overUtilized nodes num in each iteration. ( #7120 ). Contributed by Zhaobo Huang.
...
Reviewed-by: Haiyang Hu <huhaiyang926@126.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-11-02 17:06:18 +08:00
slfan1989
7a7b346b0a
Revert "HADOOP-19298. [JDK17] Add a JDK17 profile. ( #7085 ) Contributed by Shilun Fan." ( #7132 )
...
This reverts commit f931ede86b
2024-10-28 09:39:16 +08:00
Lei Yang
eb1e30395b
HDFS-17644:Add log when a node selection is rejected by BPP UpgradeDomain ( #7109 )
website / build (push) Has been cancelled
2024-10-26 17:29:43 +08:00
Syed Shameerur Rahman
0b3755347c
HADOOP-19309: S3A: CopyFromLocalFile operation fails when the source file does not contain file scheme ( #7113 )
...
Contributed by Syed Shameerur Rahman
2024-10-25 11:11:52 +01:00
Gautham B A
d1ce965645
HDFS-17636. Don't add declspec for Windows ( #7096 )
...
* Windows doesn't want the
macro _JNI_IMPORT_OR_EXPORT_
to be defined in the function
definition. It fails to compile with
the following error -
"definition of dllimport function
not allowed".
* However, Linux needs it. Hence,
we're going to add this macro
based on the OS.
* Also, we'll be compiling the `hdfs`
target as an object library so that
we can avoid linking to `jvm`
library for `get_jni_test` target.
2024-10-22 23:15:23 +05:30
Felix Nguyen
09b348753f
HDFS-17634. RBF: Fix web UI missing DN last block report ( #7080 )
2024-10-22 19:54:16 +08:00
slfan1989
f931ede86b
HADOOP-19298. [JDK17] Add a JDK17 profile. ( #7085 ) Contributed by Shilun Fan.
...
website / build (push) Has been cancelled
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Attila Doroszlai <adoroszlai@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
Reviewed-by: Min Yan <yaommen@gmail.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-18 17:16:33 +08:00
LiuGuH
6589d9f6aa
HDFS-17631. Fix RedundantEditLogInputStream.nextOp() state error when EditLogInputStream.skipUntil() throw IOException ( #7066 ). Contributed by liuguanghua.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-10-16 21:16:18 +08:00
Tao Yang
c63aafd7d1
YARN-11732. Fix potential NPE when calling SchedulerNode#reservedContainer for CapacityScheduler ( #7065 ). Contributed by Tao Yang.
...
Reviewed-by: Syed Shameerur Rahman <syedthameem1@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-10-16 21:11:31 +08:00
Davin Tjong
78a08b3b78
MAPREDUCE-7494. File stream leak when LineRecordReader is interrupted ( #7117 )
...
Contributed by Davin Tjong
2024-10-16 11:41:18 +01:00
Cheng Pan
9321e322d2
HADOOP-19310. Add JPMS options required by Java 17+ ( #7114 ) Contributed by Cheng Pan.
...
Reviewed-by: Attila Doroszlai <adoroszlai@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-16 14:15:01 +08:00
Mukund Thakur
e4b070025b
HADOOP-19291. RawLocalFileSystem to allow overlapping ranges ( #7101 )
...
ChecksumFileSystem creates the chunked ranges based on the checksum chunk size and then calls
readVectored on Raw Local which may lead to overlapping ranges in some cases.
Contributed by: Mukund Thakur
2024-10-09 08:34:47 -05:00
Steve Loughran
dc56fc385a
HADOOP-19295. S3A: large uploads can timeout over slow links ( #7089 )
...
This sets a different timeout for data upload PUT/POST calls to all
other requests, so that slow block uploads do not trigger timeouts
as rapidly as normal requests. This was always the behavior
in the V1 AWS SDK; for V2 we have to explicitly set it on the operations
we want to give extended timeouts.
Option: fs.s3a.connection.part.upload.timeout
Default: 15m
Contributed by Steve Loughran
2024-10-07 17:57:13 +01:00
Steve Loughran
50e6b49e05
HADOOP-19299. HttpReferrerAuditHeader resilience ( #7095 )
...
* HttpReferrerAuditHeader is thread safe, copying the lists/maps passed
in and using synchronized methods when necessary.
* All exceptions raised when building referrer header are caught
and swallowed.
* The first such error is logged at warn,
* all errors plus stack are logged at debug
Contributed by Steve Loughran
2024-10-07 13:53:01 +01:00
zhtttylz
1f0d9df887
HDFS-17637. Fix spotbugs in HttpFSFileSystem#getXAttr ( #7099 ) Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-06 09:16:00 +08:00
Syed Shameerur Rahman
5ea3a1bd0a
HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set (ADDENDUM) ( #7098 )
...
Contributed by Syed Shameerur Rahman
2024-10-04 14:58:53 +01:00
slfan1989
4e6432a0ab
HADOOP-19296. [JDK17] Upgrade maven-war-plugin to 3.4.0. ( #7086 ) Contributed by Shilun Fan.
...
website / build (push) Has been cancelled
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-03 22:32:11 +08:00
Susheel Gupta
1b5a2a7f65
YARN-11708: Setting maximum-application-lifetime using AQCv2 templates doesn't apply on the first submitted app ( #7041 )
2024-10-03 15:55:28 +02:00
zhtttylz
b781882020
YARN-11734. Fix spotbugs in ServiceScheduler#load ( #7088 ) Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-03 06:15:17 +08:00
Cheng Pan
3f637efaa2
HADOOP-19219. Add JPMS options required by hadoop-common ( #7084 ) Contributed by Cheng Pan.
...
website / build (push) Has been cancelled
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-10-02 10:45:25 +08:00
Syed Shameerur Rahman
e9ed21c065
HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set ( #7067 )
...
Adds new option
s3a.cross.region.access.enabled
Which is true by default
This enables cross region access as a separate config and enable/disables it irrespective of region/endpoint is set.
Contributed by Syed Shameerur Rahman
2024-10-01 20:11:11 +01:00
cxzl25
4ff0dceebd
HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider ( #7070 )
...
website / build (push) Waiting to run
Contributed by dzcxzl.
2024-10-01 14:48:48 +01:00
Steve Loughran
45b1c86fe5
HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac ( #7082 )
...
Update maven-enforcer-plugin.version to 3.5.0
Contributed by Steve Loughran
2024-10-01 14:34:05 +01:00
Sammi Chen
6fd4fea748
HADOOP-19261. Support force close a DomainSocket for server service ( #7057 )
2024-09-30 10:06:07 -07:00
Manish Bhatt
9aca73481e
HADOOP-19280. [ABFS] Initialize client timer only if metric collection is enabled ( #7061 )
...
Contributed by Manish Bhatt
2024-09-30 16:56:18 +01:00
litao
a9b7913d56
HDFS-17626. Reduce lock contention at datanode startup ( #7053 ). Contributed by Tao Li.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-09-29 10:58:13 +08:00
Ayush Saxena
3fda243419
HADOOP-19290. Operating on / in ChecksumFileSystem throws NPE. ( #7074 ). Contributed by Ayush Saxena.
2024-09-28 19:35:32 +05:30
Sarveksha Yeshavantha Raju
01401d71ef
HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI ( #7071 )
...
Replaced all LOG.info with LOG.debug
Contributed by Sarveksha Yeshavantha Raju
2024-09-27 14:20:11 +01:00
fuchaohong
3d81dde28b
HDFS-17624. Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope. ( #7042 ). Contributed by fuchaohong.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-09-27 16:56:09 +08:00
Anuj Modi
21cdb450ef
HADOOP-19284: [ABFS] Allow "fs.azure.account.hns.enabled" to be set as Account Specific Config ( #7062 )
2024-09-27 10:16:28 +05:30
Sadanand Shenoy
49a495803a
HDFS-17381. Distcp of EC files should not be limited to DFS. ( #6551 )
...
Contributed by Sadanand Shenoy
2024-09-25 17:54:09 +01:00
Syed Shameerur Rahman
21ec686be3
YARN-11702: Fix Yarn over allocating containers ( #6990 ) Contributed by Syed Shameerur Rahman.
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-09-25 12:10:15 +08:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 ( #7006 )
...
This moves Hadoop to Apache commons-collections4.
Apache commons-collections has been removed and is completely banned from the source code.
Contributed by Nihal Jain
2024-09-24 16:50:22 +01:00
Ayush Saxena
f90a703e48
HADOOP-19165. Drop protobuf 2.5.0 from the distribution ( #7051 ). Contributed by Ayush Saxena.
2024-09-24 20:58:41 +05:30
Peter Szucs
b078f86d69
YARN-11733. Fix the order of updating CPU controls with cgroup v1 ( #7069 )
2024-09-24 17:13:28 +02:00
Attila Magyar
68315744f0
HDFS-17040. Namenode web UI should set content type to application/octet-stream when uploading a file. ( #5721 )
2024-09-23 12:21:38 -07:00
Steve Loughran
37a74f0692
HADOOP-19285. [ABFS] Restore ETAGS_AVAILABLE to abfs path capabilities ( #7064 )
...
Caused by HADOOP-19131
Contributed by: Steve Loughran
2024-09-23 12:52:05 -05:00
Felix Nguyen
fccc268cde
HADOOP-19283. Move all DistCp execution logic to execute() ( #7060 )
...
Co-authored-by: Felix Nguyen <kokonguyen191@gmail.com>
2024-09-23 15:39:56 +08:00
gavin.wang
f5345b881d
YARN-11560. Fix NPE bug when multi-node enabled with schedule asynchronously ( #6021 ). Contributed by wangzhongwei.
2024-09-23 13:09:15 +05:30
cxzl25
ae63d8edd5
HDFS-17621. Make PathIsNotEmptyDirectoryException terse ( #7036 ). Contributed by dzcxzl.
2024-09-23 13:07:30 +05:30
dannytbecker
50fcfd8b29
HDFS-17526. getMetadataInputStream should use getShareDeleteFileInputStream for windows ( #6826 ). Contributed by Danny Becker
2024-09-23 12:16:25 +05:30
Arjun Mohnot
d8ca2dbe34
YARN-11730. Mark unreported nodes as LOST on RM Startup/HA failover ( #7049 ) Contributed by Arjun Mohnot.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-09-23 10:46:20 +08:00
huhaiyang
81faae6343
Revert "HDFS-17611. Move all DistCp execution logic to execute() ( #7025 )" ( #7059 )
...
This reverts commit ff75aa52f3
2024-09-23 10:44:14 +08:00
Felix Nguyen
ff75aa52f3
HDFS-17611. Move all DistCp execution logic to execute() ( #7025 )
...
---------
Co-authored-by: Felix Nguyen <kokonguyen191@gmail.com>
2024-09-23 10:36:20 +08:00
Ayush Saxena
28538d628e
HADOOP-19164. Hadoop CLI MiniCluster is broken ( #7050 ). Contributed by Ayush Saxena.
...
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
2024-09-21 21:26:51 +05:30
manika137
6bcc254123
HADOOP-19279. ABFS: Disabling Apache Http Client as Default Http Client for ABFS Driver( #7055 )
...
As part of work done under HADOOP-19120 [ABFS]: ApacheHttpClient adaptation as network library - ASF JIRA
Apache HTTP Client was introduced as an alternative Network Library that can be used with ABFS Driver. Earlier JDK Http Client was the only supported network library.
Apache HTTP Client was found to be more helpful in terms of controls and knobs it provides to manage the Network aspects of the driver better. Hence, the default Network Client was made to be used with the ABFS Driver.
Recently while running scale workloads, we observed a regression where some unexpected wait time was observed while establishing connections. A possible fix has been identified and we are working on getting it fixed.
There was also a possible NPE scenario was identified on the new network client code.
Until we are done with the code fixes and revalidated the whole Apache client flow, we would like to make JDK Client as default client again. The support will still be there, but it will be disabled behind a config.
Contributed by: manika137
2024-09-20 16:38:56 -05:00
