Alex
5c7e40f910
HADOOP-19111. Removing redundant debug message about client info ( #6666 ). Contributed by Zhongkun Wu.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-25 11:44:33 +08:00
yu liang
55dca911cc
HADOOP-19052.Hadoop use Shell command to get the count of the hard link which takes a lot of time ( #6587 ) Contributed by liangyu.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-24 10:14:27 +08:00
LiuGuH
a60b5e2de3
MAPREDUCE-7469. NNBench createControlFiles should use thread pool to improve performance. ( #6463 ) Contributed by liuguanghua.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-23 06:15:56 +08:00
huhaiyang
8cd4704e0a
HDFS-17430. RecoveringBlock will skip no live replicas when get block recovery command. ( #6635 )
2024-03-22 09:43:12 -04:00
XiaobaoWu
062836c020
HDFS-17436. Supplement log information for AccessControlException ( #6651 )
2024-03-22 11:21:03 +08:00
XiaobaoWu
a375ef8cfa
YARN-11626. Optimize ResourceManager's operations on Zookeeper metadata ( #6616 )
...
Co-authored-by: wuxiaobao <xbaowu@163.com>
2024-03-21 03:12:14 -04:00
Takanobu Asanuma
adab3a22aa
HDFS-17432. Fix junit dependency to enable JUnit4 tests to run in hadoop-hdfs-rbf ( #6639 )
...
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2024-03-21 14:53:02 +09:00
hfutatzhanghb
2eb7246ea7
HDFS-17433. metrics sumOfActorCommandQueueLength should only record valid commands. ( #6644 )
2024-03-20 23:41:35 -04:00
huhaiyang
77c600d769
HDFS-17426. Remove Invalid FileSystemECReadStats logic in DFSInputStream ( #6628 )
2024-03-21 10:33:41 +08:00
Peter Szucs
a957cd5049
YARN-5305. Allow log aggregation to discard expired delegation tokens ( #6625 )
2024-03-20 15:33:10 +01:00
huhaiyang
12a26d8b19
HDFS-17431. Fix log format for BlockRecoveryWorker#recoverBlocks ( #6643 )
2024-03-19 23:22:45 -04:00
Adnan Hemani
8b2058a4e7
HADOOP-19050. S3A: Support S3 Access Grants ( #6544 )
...
This adds support for Amazon S3 Access Grants to the S3A connector.
For more information, see:
* https://aws.amazon.com/s3/features/access-grants/
* https://github.com/aws/aws-s3-accessgrants-plugin-java-v2/
Contributed by Adnan Hemani
2024-03-19 17:49:51 +00:00
Steve Loughran
705fb8323b
HADOOP-19119. Spotbugs: possible NPE in org.apache.hadoop.crypto.key.kms.ValueQueue.getSize() ( #6642 )
...
Spotbugs is mistaken here as it doesn't observer the read/write locks used
to manage exclusive access to the maps.
* cache the value between checks
* tag as @VisibleForTesting
Contributed by Steve Loughran
2024-03-19 17:18:07 +00:00
slfan1989
ff3f2255d2
HADOOP-19112. Hadoop 3.4.0 release wrap-up. ( #6640 ) Contributed by Shilun Fan.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-19 20:08:03 +08:00
Vinayakumar B
0f51d2a4ec
HADOOP-14451. Deadlock in NativeIO ( #6632 )
2024-03-18 10:53:21 +05:30
Tsz-Wo Nicholas Sze
b25b28e5bb
HDFS-17380. FsImageValidation: remove inaccessible nodes. ( #6549 ). Contributed by Tsz-wo Sze.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-18 11:52:17 +08:00
drankye
4d88f9892a
HADOOP-19085. Compatibility Benchmark over HCFS Implementations
...
Contributed by Han Liu
2024-03-17 16:48:29 +08:00
Junfan Zhang
783cc3eda0
YARN-11660. Fix performance regression for SingleConstraintAppPlacementAllocator ( #6623 ) Contributed by Junfan Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-03-15 08:14:38 +08:00
Viraj Jasani
a325876fec
HADOOP-19066. Run FIPS test for valid bucket locations (ADDENDUM) ( #6624 )
...
FIPS is only supported in north america AWS regions; relevant tests in
ITestS3AEndpointRegion are skipped for buckets with different endpoints/regions.
2024-03-13 13:21:50 +00:00
PJ Fanning
e28c78f9a2
HADOOP-19088. Use jersey-json 1.22.0 ( #6585 )
...
Contributed by pjfanning
2024-03-12 20:16:47 +00:00
Viraj Jasani
44c14edac7
HADOOP-19066. S3A: AWS SDK V2 - Enabling FIPS should be allowed with central endpoint ( #6539 )
...
Contributed by Viraj Jasani
2024-03-12 18:49:06 +00:00
Lei313
e211f6f83d
HDFS-17391. Adjust the checkpoint io buffer size to the chunk size ( #6594 ). Contributed by lei w.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-12 18:36:43 +08:00
Lei313
dbf08c872a
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin ( #6621 ). Contributed by lei w and Hualong Zhang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-03-12 12:25:33 +08:00
Steve Loughran
bb32aec88e
HADOOP-19043. S3A: Regression: ITestS3AOpenCost fails on prefetch test runs ( #6465 )
...
Disables the new tests added in:
HADOOP-19027. S3A: S3AInputStream doesn't recover from HTTP/channel exceptions #6425
The underlying issue here is that the block prefetch code can identify
when there's a mismatch between declared and actual length, and doesn't
store any of the incomplete buffer.
This should be addressed in HADOOP-18184.
Contributed by Steve Loughran
2024-03-08 12:48:38 +00:00
PJ Fanning
fc166d3aec
HADOOP-19090. Use protobuf-java 3.23.4. ( #6593 ). Contributed by PJ Fanning.
2024-03-07 15:09:01 +05:30
ritegarg
58afe43769
HDFS-17299. Adding rack failure tolerance when creating a new file ( #6566 )
2024-03-06 13:08:05 -08:00
hfutatzhanghb
7012986fc3
HDFS-17345. Add a metrics to record block report generating cost time. ( #6475 ). Contributed by farmmamba.
...
Reviewed-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-03-06 16:59:00 +08:00
dannytbecker
5584efd8d4
HDFS-17396. BootstrapStandby should download rollback image during RollingUpgrade ( #6583 )
2024-03-05 11:31:31 -08:00
Jtdellaringa
a2d7241190
HDFS-17404. Add NN Socket Address to log when processing command from active NN ( #6598 )
...
* Add nnSocketAddress to log when processing command from active nn
2024-03-05 10:33:22 -08:00
Steve Loughran
dae871e3e0
YARN-11657. Remove protobuf-2.5 from hadoop-yarn-api module ( #6575 ) ( #6580 )
...
The import of protobuf-java-2.5 in the hadoop-yarn-api module
is downgraded from "compile" to "provided"
This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar
It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar
Contributed by Steve Loughran
2024-03-05 11:01:14 +00:00
HarshitGupta11
d974a12f39
HADOOP-19082: S3A: Update AWS SDK V2 to 2.24.6 ( #6568 )
...
Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module.
Contributed by Harshit Gupta
2024-03-05 10:15:05 +00:00
Steve Loughran
095229fefb
HADOOP-19097. S3A: Set fs.s3a.connection.establish.timeout to 30s ( #6601 )
...
This is consistent with the value in the hadoop-aws source code
Contributed by Steve Loughran
2024-03-05 10:10:27 +00:00
Jian Zhang
a6aa2925fb
HDFS-17333. DFSClient supports lazy resolution from hostname to IP. ( #6430 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-03-02 21:35:24 +09:00
Steve Loughran
a0ce2170db
HADOOP-19084. Prune hadoop-common transitive dependencies ( #6574 ) ( #6582 )
...
Exclude more artifacts which are dependencies of hadoop-* modules,
with the goal of keeping conflict out of downstream applications.
In particular we have pruned the dependencies of of:
-zookeeper
-other libraries referencing logging
This keeps slf4j-log4j12 and log4j12 off the classpath
of applications importing hadoop-common.
Somehow logback references do still surface; applications
pulling in hadoop-common directly or indirectly should
review their imports carefully.
Contributed by Steve Loughran
2024-03-01 12:51:13 +00:00
huhaiyang
34bb28649c
HDFS-17406. Suppress UnresolvedPathException in hdfs router log ( #6603 )
2024-03-01 19:03:41 +08:00
hfutatzhanghb
15af52954f
HDFS-17358. EC: infinite lease recovery caused by the length of RWR equals to zero or datanode does not have the replica. ( #6509 ). Contributed by farmmamba.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-02-27 10:19:57 +08:00
ZanderXu
a897e745f5
HDFS-17393. Remove unused FSNamesytemLock cond in FSNamesystem ( #6567 )
2024-02-23 15:37:27 +08:00
slfan1989
10ab8abccd
Revert "HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. ( #6537 )" ( #6578 )
...
This reverts commit 555faf28ce
2024-02-23 14:25:15 +08:00
Anuj Modi
99b9e7fb43
HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport ( #6069 )
...
Contributed By: Anuj Modi
2024-02-22 11:49:37 -06:00
huangzhaobo
6dfebdbce4
HADOOP-19065. Update Protocol Buffers installation to 3.21.12 ( #6526 )
...
Instructions and docker script for installing protobuf updated from 3.7.1 to 3.21.12
Contributed by huangzhaobo
2024-02-22 17:09:46 +00:00
Lars Francke
d1daf26b85
Fixes HDFS-17181 by routing all CREATE requests to the BlockManager ( #6108 )
2024-02-20 10:58:49 -08:00
Anuj Modi
1336c362e5
Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. ( #5881 )
...
Contributed By: Anuj Modi
2024-02-20 11:31:42 -06:00
huhaiyang
03d9acaa86
HDFS-17361. DiskBalancer: Query command support with multiple nodes ( #6508 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-19 09:34:59 +09:00
zhtttylz
9751b6e41a
HDFS-17146.Use the dfsadmin -reconfig command to initiate reconfiguration on all decommissioning datanodes. ( #6504 ) Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-18 09:15:40 +08:00
Lei313
12498b35bb
HDFS-17354:Delay invoke clearStaleNamespacesInRouterStateIdContext during router start up ( #6498 )
2024-02-13 16:24:23 -08:00
Steve Loughran
095dfcca30
HADOOP-18088. Replace log4j 1.x with reload4j. ( #4052 )
...
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Includes HADOOP-18354. Upgrade reload4j to 1.22.2 due to XXE vulnerability (#4607 ).
Log4j 1.2.17 has been replaced by reloadj 1.22.2
SLF4J is at 1.7.36
2024-02-13 16:33:51 +00:00
Steve Loughran
7651afd3db
HADOOP-19057. S3A: Landsat bucket used in tests no longer accessible ( #6515 )
...
The AWS landsat data previously used in some S3A tests is no
longer accessible
This PR moves to the new external file
s3a://noaa-cors-pds/raw/2024/001/akse/AKSE001x.24_.gz
* Large enough file for scale tests
* Bucket supports anonymous access
* Ends in .gz to keep codec tests happy
* No spaces in path to keep bucket-info happy
Test Code Changes
* Leaves the test key name alone: fs.s3a.scale.test.csvfile
* Rename all methods and fields move remove "csv" from their names and
move to "external file" we no longer require it to be CSV.
* Path definition and helper methods have been moved to PublicDatasetTestUtils
* Improve error reporting in ITestS3AInputStreamPerformance if the file
is too short
With S3 Select removed, there is no need for the file to be
a CSV file; there is a test which tries to unzip it; other
tests have a minimum file size.
Consult the JIRA for the settings to add to auth-keys.xml
to switch earlier builds to this same file.
Contributed by Steve Loughran
2024-02-13 10:46:36 +00:00
Takanobu Asanuma
5cbe52f4e8
HDFS-17362. RBF: Implement RouterObserverReadConfiguredFailoverProxyProvider ( #6510 )
...
Co-authored-by: Chunyi Yang <cyang@lycorp.co.jp>
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
2024-02-13 10:49:39 +09:00
slfan1989
555faf28ce
HADOOP-19071. Update maven-surefire-plugin from 3.0.0 to 3.2.5. ( #6537 ) Contributed by Shilun Fan
...
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-11 07:41:46 +08:00
Sadanand Shenoy
0bf439c0f9
HDFS-17376. Distcp creates Factor 1 replication file on target if Source is EC. ( #6540 )
2024-02-09 17:26:32 +00:00
