big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
27,469 Commits 2 Branches 2 Tags 264 MiB
9a743bd17f
Commit Graph

27469 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dominik Diedrich
9a743bd17f
HADOOP-19315. Upgrade Apache Avro to 1.11.4 (#7128) 
* All field access is now via setter/getter methods
* To use Avro to marshal Serializable objects,
  the packages they are in must be declared in the system property
  "org.apache.avro.SERIALIZABLE_PACKAGES"
  
This is required to address
- CVE-2024-47561
- CVE-2023-39410  

This change is not backwards compatible.

Contributed by Dominik Diedrich
 2024-11-11 15:46:36 +00:00
Joseph Dell'Aringa
9657276492
HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (#7127) 
HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (#7127)

---------

Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
 2024-11-07 10:15:51 -08:00
Anuj Modi
487727a5d1
HADOOP-18960: [ABFS] Making Contract tests run in sequential and Other Test Fixes (#7104) 
Contributed by: Anuj Modi
 2024-11-05 16:12:03 -06:00
muskan1012
f7651e2f63
HADOOP-19243. Upgrade Mockito version to 4.11.0 (#6968) 
Mockito is now at a JDK-17 compatible version.

Contributed by Muskan Mishra
 2024-11-05 17:35:53 +00:00
Sebastian Klemke
51ebc3c20e
HADOOP-18583. Fix loading of OpenSSL 3.x symbols (#5256) 
Contributed by Sebastian Klemke
 2024-11-05 17:14:36 +00:00
yanmin
9ae01bdbe8
HADOOP-19143. Upgrade commons-cli to 1.9.0 (#7126) Contributed by Min Yan. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-11-05 10:39:49 +08:00
yanmin
df979e70de
HADOOP-19297. [JDK17] Upgrade maven.plugin-tools.version to 3.10.2 (#7125) Contributed by Min Yan. 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-11-05 07:46:35 +08:00
Christos Bisias
66baf1eb51
HADOOP-18682. Move hadoop docker scripts under the main source code (#6483). Contributed by Christos Bisias. 2024-11-04 22:22:37 +05:30
Lei313
e4789a2fd3
HDFS-17607. Reduce the number of times conf is loaded when DataNode startUp (#7012). Contributed by lei w. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-04 20:10:08 +08:00
hfutatzhanghb
4f3abd2f48
HDFS-17654. Fix bugs in TestRouterMountTable (#7137). Contributed by farmmamba. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-02 17:27:06 +08:00
Zhaobo Huang
00cddf5bea
HDFS-17646. Add Option to limit Balancer overUtilized nodes num in each iteration. (#7120). Contributed by Zhaobo Huang. 
Reviewed-by: Haiyang Hu <huhaiyang926@126.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-11-02 17:06:18 +08:00
slfan1989
7a7b346b0a
Revert "HADOOP-19298. [JDK17] Add a JDK17 profile. (#7085) Contributed by Shilun Fan." (#7132) 
This reverts commit f931ede86b.
 2024-10-28 09:39:16 +08:00
Lei Yang
eb1e30395b
HDFS-17644:Add log when a node selection is rejected by BPP UpgradeDomain (#7109)
Some checks failed
website / build (push) Has been cancelled
Details
2024-10-26 17:29:43 +08:00
Syed Shameerur Rahman
0b3755347c
HADOOP-19309: S3A: CopyFromLocalFile operation fails when the source file does not contain file scheme (#7113) 
Contributed by Syed Shameerur Rahman
 2024-10-25 11:11:52 +01:00
Gautham B A
d1ce965645
HDFS-17636. Don't add declspec for Windows (#7096) 
* Windows doesn't want the
  macro _JNI_IMPORT_OR_EXPORT_
  to be defined in the function
  definition. It fails to compile with
  the following error -
  "definition of dllimport function
  not allowed".
* However, Linux needs it. Hence,
  we're going to add this macro
  based on the OS.
* Also, we'll be compiling the `hdfs`
  target as an object library so that
  we can avoid linking to `jvm`
  library for `get_jni_test` target.
 2024-10-22 23:15:23 +05:30
Felix Nguyen
09b348753f
HDFS-17634. RBF: Fix web UI missing DN last block report (#7080) 2024-10-22 19:54:16 +08:00
slfan1989
f931ede86b
HADOOP-19298. [JDK17] Add a JDK17 profile. (#7085) Contributed by Shilun Fan.
Some checks failed
website / build (push) Has been cancelled
Details 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Attila Doroszlai <adoroszlai@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
Reviewed-by: Min Yan <yaommen@gmail.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-18 17:16:33 +08:00
LiuGuH
6589d9f6aa
HDFS-17631. Fix RedundantEditLogInputStream.nextOp() state error when EditLogInputStream.skipUntil() throw IOException (#7066). Contributed by liuguanghua. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-10-16 21:16:18 +08:00
Tao Yang
c63aafd7d1
YARN-11732. Fix potential NPE when calling SchedulerNode#reservedContainer for CapacityScheduler (#7065). Contributed by Tao Yang. 
Reviewed-by: Syed Shameerur Rahman <syedthameem1@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-10-16 21:11:31 +08:00
Davin Tjong
78a08b3b78
MAPREDUCE-7494. File stream leak when LineRecordReader is interrupted (#7117) 
Contributed by Davin Tjong
 2024-10-16 11:41:18 +01:00
Cheng Pan
9321e322d2
HADOOP-19310. Add JPMS options required by Java 17+ (#7114) Contributed by Cheng Pan. 
Reviewed-by: Attila Doroszlai <adoroszlai@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-16 14:15:01 +08:00
Mukund Thakur
e4b070025b
HADOOP-19291. RawLocalFileSystem to allow overlapping ranges (#7101) 
ChecksumFileSystem creates the chunked ranges based on the checksum chunk size and then calls
readVectored on Raw Local which may lead to overlapping ranges in some cases.

Contributed by: Mukund Thakur
 2024-10-09 08:34:47 -05:00
Steve Loughran
dc56fc385a
HADOOP-19295. S3A: large uploads can timeout over slow links (#7089) 
This sets a different timeout for data upload PUT/POST calls to all
other requests, so that slow block uploads do not trigger timeouts
as rapidly as normal requests. This was always the behavior
in the V1 AWS SDK; for V2 we have to explicitly set it on the operations
we want to give extended timeouts. 

Option:  fs.s3a.connection.part.upload.timeout
Default: 15m

Contributed by Steve Loughran
 2024-10-07 17:57:13 +01:00
Steve Loughran
50e6b49e05
HADOOP-19299. HttpReferrerAuditHeader resilience (#7095) 
* HttpReferrerAuditHeader is thread safe, copying the lists/maps passed
  in and using synchronized methods when necessary.
* All exceptions raised when building referrer header are caught
  and swallowed.
* The first such error is logged at warn,  
* all errors plus stack are logged at debug

Contributed by Steve Loughran
 2024-10-07 13:53:01 +01:00
zhtttylz
1f0d9df887
HDFS-17637. Fix spotbugs in HttpFSFileSystem#getXAttr (#7099) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-06 09:16:00 +08:00
Syed Shameerur Rahman
5ea3a1bd0a
HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set (ADDENDUM) (#7098) 
Contributed by Syed Shameerur Rahman
 2024-10-04 14:58:53 +01:00
slfan1989
4e6432a0ab
HADOOP-19296. [JDK17] Upgrade maven-war-plugin to 3.4.0. (#7086) Contributed by Shilun Fan.
Some checks failed
website / build (push) Has been cancelled
Details 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-03 22:32:11 +08:00
Susheel Gupta
1b5a2a7f65
YARN-11708: Setting maximum-application-lifetime using AQCv2 templates doesn't apply on the first submitted app (#7041) 2024-10-03 15:55:28 +02:00
zhtttylz
b781882020
YARN-11734. Fix spotbugs in ServiceScheduler#load (#7088) Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-03 06:15:17 +08:00
Cheng Pan
3f637efaa2
HADOOP-19219. Add JPMS options required by hadoop-common (#7084) Contributed by Cheng Pan.
Some checks failed
website / build (push) Has been cancelled
Details 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-10-02 10:45:25 +08:00
Syed Shameerur Rahman
e9ed21c065
HADOOP-19286: S3A: Support cross region access when S3 region/endpoint is set (#7067) 
Adds new option
   s3a.cross.region.access.enabled
Which is true by default

This enables cross region access as a separate config and enable/disables it irrespective of region/endpoint is set.

Contributed by Syed Shameerur Rahman
 2024-10-01 20:11:11 +01:00
cxzl25
4ff0dceebd
HADOOP-19288. hadoop-client-runtime to exclude dnsjava InetAddressResolverProvider (#7070)
Some checks are pending
website / build (push) Waiting to run
Details 
Contributed by dzcxzl.
 2024-10-01 14:48:48 +01:00
Steve Loughran
45b1c86fe5
HADOOP-19294. NPE on maven enforcer with -Pnative on arm mac (#7082) 
Update maven-enforcer-plugin.version to 3.5.0

Contributed by Steve Loughran
 2024-10-01 14:34:05 +01:00
Sammi Chen
6fd4fea748
HADOOP-19261. Support force close a DomainSocket for server service (#7057) 2024-09-30 10:06:07 -07:00
Manish Bhatt
9aca73481e
HADOOP-19280. [ABFS] Initialize client timer only if metric collection is enabled (#7061) 
Contributed by Manish Bhatt
 2024-09-30 16:56:18 +01:00
litao
a9b7913d56
HDFS-17626. Reduce lock contention at datanode startup (#7053). Contributed by Tao Li. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-29 10:58:13 +08:00
Ayush Saxena
3fda243419
HADOOP-19290. Operating on / in ChecksumFileSystem throws NPE. (#7074). Contributed by Ayush Saxena. 2024-09-28 19:35:32 +05:30
Sarveksha Yeshavantha Raju
01401d71ef
HADOOP-19281. MetricsSystemImpl should not print INFO message in CLI (#7071) 
Replaced all LOG.info with LOG.debug

Contributed by Sarveksha Yeshavantha Raju
 2024-09-27 14:20:11 +01:00
fuchaohong
3d81dde28b
HDFS-17624. Fix DFSNetworkTopology#chooseRandomWithStorageType() availableCount when excluded node is not in selected scope. (#7042). Contributed by fuchaohong. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2024-09-27 16:56:09 +08:00
Anuj Modi
21cdb450ef
HADOOP-19284: [ABFS] Allow "fs.azure.account.hns.enabled" to be set as Account Specific Config (#7062) 2024-09-27 10:16:28 +05:30
Sadanand Shenoy
49a495803a
HDFS-17381. Distcp of EC files should not be limited to DFS. (#6551) 
Contributed by Sadanand Shenoy
 2024-09-25 17:54:09 +01:00
Syed Shameerur Rahman
21ec686be3
YARN-11702: Fix Yarn over allocating containers (#6990) Contributed by Syed Shameerur Rahman. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2024-09-25 12:10:15 +08:00
Nihal Jain
e602c601dd
HADOOP-15760. Upgrade commons-collections to commons-collections4 (#7006) 
This moves Hadoop to Apache commons-collections4.

Apache commons-collections has been removed and is completely banned from the source code.

Contributed by Nihal Jain
 2024-09-24 16:50:22 +01:00
Ayush Saxena
f90a703e48
HADOOP-19165. Drop protobuf 2.5.0 from the distribution (#7051). Contributed by Ayush Saxena. 2024-09-24 20:58:41 +05:30
Peter Szucs
b078f86d69
YARN-11733. Fix the order of updating CPU controls with cgroup v1 (#7069) 2024-09-24 17:13:28 +02:00
Attila Magyar
68315744f0
HDFS-17040. Namenode web UI should set content type to application/octet-stream when uploading a file. (#5721) 2024-09-23 12:21:38 -07:00
Steve Loughran
37a74f0692
HADOOP-19285. [ABFS] Restore ETAGS_AVAILABLE to abfs path capabilities (#7064) 
Caused by HADOOP-19131  

Contributed by: Steve Loughran
 2024-09-23 12:52:05 -05:00
Felix Nguyen
fccc268cde
HADOOP-19283. Move all DistCp execution logic to execute() (#7060) 
Co-authored-by: Felix Nguyen <kokonguyen191@gmail.com>
 2024-09-23 15:39:56 +08:00
gavin.wang
f5345b881d
YARN-11560. Fix NPE bug when multi-node enabled with schedule asynchronously (#6021). Contributed by wangzhongwei. 2024-09-23 13:09:15 +05:30
cxzl25
ae63d8edd5
HDFS-17621. Make PathIsNotEmptyDirectoryException terse (#7036). Contributed by dzcxzl. 2024-09-23 13:07:30 +05:30