Sadanand Shenoy
0bf439c0f9
HDFS-17376. Distcp creates Factor 1 replication file on target if Source is EC. ( #6540 )
2024-02-09 17:26:32 +00:00
Adnan Hemani
50d256ef3c
HADOOP-19059. S3A: Update AWS Java SDK to 2.23.19 ( #6538 )
...
Contributed by Adnan Hemani
2024-02-08 20:38:37 +00:00
slfan1989
8011b21c52
HADOOP-19069. Use hadoop-thirdparty 1.2.0. ( #6533 ) Contributed by Shilun Fan
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-08 19:18:04 +08:00
Steve Loughran
3f98cb6741
HADOOP-19045. CreateSession Timeout - followup ( #6532 )
...
This is a followup to PR:
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470 )
Remove all declarations of fs.s3a.connection.request.timeout
in
- hadoop-common/src/main/resources/core-default.xml
- hadoop-aws/src/test/resources/core-site.xml
New test in TestAwsClientConfig to verify that the value
defined in fs.s3a.Constants class is used.
This is brittle to someone overriding it in their test setups,
but as this test is intended to verify that the option is not
explicitly set, there's no workaround.
Contributed by Steve Loughran
2024-02-07 12:07:54 +00:00
huhaiyang
5ad7737132
HDFS-17342. Fix DataNode may invalidates normal block causing missing block ( #6464 ). Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Reviewed-by: Chengwei Wang <1139557635@qq.com>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-02-06 17:52:52 +08:00
Takanobu Asanuma
9a7eeadaac
HDFS-17370. Fix junit dependency for running parameterized tests in hadoop-hdfs-rbf ( #6522 )
2024-02-06 11:54:14 +09:00
Antonio Murgia
b11159d799
HADOOP-18993. Add option fs.s3a.classloader.isolation ( #6301 )
...
The option fs.s3a.classloader.isolation (default: true) can be set to false to disable s3a classloader isolation;
This can assist in using custom credential providers and other extension points.
Contributed by Antonio Murgia
2024-02-05 17:59:36 +00:00
Jia Fan
4f0f5a546c
HADOOP-19049. Fix StatisticsDataReferenceCleaner classloader leak ( #6488 )
...
Contributed by Jia Fan
2024-02-03 14:48:52 +00:00
Eremenko Valentin
141627778d
YARN-11362: Fix several typos in YARN codebase of misspelled resource ( #6474 ) Contributed by EremenkoValentin.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:34:42 +08:00
Alexander
20d8596af2
HDFS-17353. Fix failing RBF module tests. ( #6491 ) Contributed by Alexander Bogdanov
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:26:30 +08:00
huhaiyang
6464507cd1
HDFS-17369. Add uuid into datanode info for NameNodeMXBean ( #6521 ) Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:20:04 +08:00
Xing Lin
d74e5160cd
HADOOP-19061 Capture exception from rpcRequestSender.start() in IPC.Connection.run() ( #6519 )
...
* HADOOP-19061 - Capture exception from rpcRequestSender.start() in IPC.Connection.run() and proper cleaning is followed if an exception is thrown.
---------
Co-authored-by: Xing Lin <xinglin@linkedin.com>
2024-02-02 16:22:16 -08:00
Viraj Jasani
d278b349f6
HADOOP-19044. S3A: AWS SDK V2 - Update region logic ( #6479 )
...
Improves region handling in the S3A connector, including enabling cross-region support
when that is considered necessary.
Consult the documentation in connecting.md/connecting.html for the current
resolution process.
Contributed by Viraj Jasani
2024-02-02 17:07:05 +00:00
Viraj Jasani
7504b8505f
HADOOP-18980. S3A credential provider remapping: make extensible ( #6406 )
...
Contributed by Viraj Jasani
2024-02-02 17:02:48 +00:00
DieterDP
be13e94843
HADOOP-18987. Various fixes to FileSystem API docs ( #6292 )
...
Contributed by Dieter De Paepe
2024-02-02 11:49:31 +00:00
hfutatzhanghb
4f4b846986
HDFS-17359. EC: recheck failed streamers should only after flushing all packets. ( #6503 ). Contributed by farmmamba.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-01 23:53:37 +09:00
Takanobu Asanuma
2f1718c363
HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. ( #6512 ) Contributed by Takanobu Asanuma.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-31 13:30:35 +08:00
Yang Jiandan
98259fb4da
YARN-11653. Add Totoal_Memory and Total_Vcores columns in Nodes page ( #6501 ) Contributed by Jiandan Yang.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-31 07:07:54 +08:00
Steve Loughran
8261229daa
HADOOP-18830. Cut S3 Select ( #6144 )
...
Cut out S3 Select
* leave public/unstable constants alone
* s3guard tool will fail with error
* s3afs. path capability will fail
* openFile() will fail with specific error
* s3 select doc updated
* Cut eventstream jar
* New test: ITestSelectUnsupported verifies new failure
handling above
Contributed by Steve Loughran
2024-01-30 16:12:27 +00:00
Steve Loughran
6da1a19a83
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation ( #6470 )
...
New test ITestCreateSessionTimeout to verify that the duration set
in fs.s3a.connection.request.timeout is passed all the way down.
This is done by adding a sleep() in a custom signer and verifying
that it is interrupted and that an AWSApiCallTimeoutException is
raised.
+ Fix testRequestTimeout()
* doesn't skip if considered cross-region
* sets a minimum duration of 0 before invocation
* resets the minimum afterwards
Contributed by Steve Loughran
2024-01-30 15:32:24 +00:00
Yang Jiandan
dffb505725
YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue ( #6490 ) Contributed by Jiandan Yang.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-29 21:37:32 +08:00
Peter Szucs
4046751939
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup ( #6499 )
2024-01-26 14:42:59 +01:00
Benjamin Teke
897f446d54
HADOOP-19051: Highlight Capacity Scheduler new features in release for the release 3.4.0 ( #6500 ) Contributed by Benjamin Teke.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-26 13:33:55 +08:00
Tsz-Wo Nicholas Sze
da34ecdb83
HADOOP-19035. CrcUtil/CrcComposer should not throw IOException for non-IO. ( #6443 )
2024-01-25 10:35:32 -08:00
Lei313
ac471d7daa
HDFS-17339:Skip cacheReport when one blockPool does not have CacheBlock on this DataNode ( #6456 ). Contributed by lei w.
...
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 21:15:20 +08:00
Tamas Domok
7272a3d960
YARN-11641. Fix getCapacityConfigurationTypeForQueues for absolute mode with zero resource. ( #6435 )
2024-01-25 13:58:00 +01:00
slfan1989
38f10c657e
HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements. ( #6462 ) Contributed by Shilun Fan.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-25 15:42:21 +08:00
huhaiyang
caba9bbab3
HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt ( #6476 ). Contributed by Haiyang Hu.
...
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 12:49:18 +08:00
slfan1989
6b80b1e60f
Revert "YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup" ( #6497 )
2024-01-25 11:44:15 +08:00
Peter Szucs
dd72e8fbc3
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup ( #5332 )
2024-01-24 15:51:07 +01:00
Tamas Domok
af5c8e70ea
YARN-11645. Order the queues by queuePath in json assert tests. ( #6432 )
2024-01-24 10:12:24 +01:00
LiuGuH
8193a84020
HDFS-17324. RBF: Router should not return nameservices that not enable observer r… ( #6412 )
2024-01-23 12:30:08 -08:00
jialiang
b2fac14828
HADOOP-19019: Parallel Maven Build Support for Apache Hadoop ( #6373 ). Contributed by JiaLiangC.
...
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-01-23 14:51:20 +08:00
Ferenc Erdelyi
8243da8cb0
YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy ( #6455 )
2024-01-22 15:41:48 +01:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. ( #6368 ). Contributed by farmmamba.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-22 11:50:51 +08:00
Pranav Saxena
7dc166ddc7
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls ( #6022 )
...
Address JDK bug JDK-8314978 related to handling of HTTP 100
responses.
https://bugs.openjdk.org/browse/JDK-8314978
In the AbfsHttpOperation, after sendRequest() we call processResponse()
method from AbfsRestOperation.
Even if the conn.getOutputStream() fails due to expect-100 error,
we consume the exception and let the code go ahead.
This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong() after
getOutputStream() has failed. These invocation all lead to server calls.
This commit aims to prevent this.
If connection.getOutputStream() fails due to an Expect-100 error,
the ABFS client does not invoke getHeaderField(), getHeaderFields(),
getHeaderFieldLong() or getInputStream().
getResponseCode() is safe as on the failure it sets the
responseCode variable in HttpUrlConnection object.
Contributed by Pranav Saxena
2024-01-21 19:14:54 +00:00
Steve Loughran
d274f778c1
HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 ( #6467 )
...
This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down
to calls to CreateSession made in the AWS SDK to get S3 Express session tokens.
Contributed by Steve Loughran
2024-01-21 19:00:34 +00:00
PJ Fanning
76691dfa14
HADOOP-18894: upgrade sshd-core due to CVEs ( #6060 ) Contributed by PJ Fanning.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-21 08:13:25 +08:00
LiuGuH
2a1ee8dfcd
HDFS-17311. RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue. ( #6392 ) Contributed by liuguanghua.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:55:23 +08:00
slfan1989
15e1789baf
Revert "HDFS-16016. BPServiceActor to provide new thread to handle IBR ( #2998 )" ( #6457 ) Contributed by Shilun Fan.
...
This reverts commit c1bf3cb0
2024-01-20 07:51:55 +08:00
Susheel Gupta
d0df0689b4
YARN-11607: TestTimelineAuthFilterForV2 fails intermittently ( #6459 ) Contributed by Susheel Gupta.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:42:08 +08:00
Jian Zhang
1036544480
HDFS-17302. RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation. ( #6380 )
2024-01-19 14:02:21 -08:00
slfan1989
8444f69511
Preparing for 3.5.0 development ( #6411 )
...
Co-authored-by: slfan1989 <slfan1989@apache.org>
2024-01-19 15:05:22 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException ( #6446 )
...
Print a warn log message for read retries and only print the full stack trace for a read request failure.
Contributed by: Xing Lin
2024-01-18 18:03:28 -08:00
Lei313
cc4c4be1b7
HDFS-17331:Fix Blocks are always -1 and DataNode version are always UNKNOWN in federationhealth.html ( #6429 ). Contributed by lei w.
...
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-18 21:10:54 +08:00
slfan1989
4c3d4e6a57
HADOOP-19038. Improve create-release RUN script. ( #6448 ) Contributed by Shilun Fan.
...
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-18 19:12:12 +08:00
PJ Fanning
04e447cfa7
YARN-11647. use StandardCharsets.UTF_8 ( #6447 ) Contributed by PJ Fanning.
...
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-18 13:53:18 +08:00
hfutatzhanghb
ba6ada73ac
HDFS-17337. RPC RESPONSE time seems not exactly accurate when using FSEditLogAsync. ( #6439 ). Contributed by farmmamba.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-18 11:10:05 +08:00
Steve Loughran
eeb657e85f
HADOOP-19033. S3A: disable checksums when fs.s3a.checksum.validation = false ( #6441 )
...
Add new option fs.s3a.checksum.validation, default false, which
is used when creating s3 clients to enable/disable checksum
validation.
When false, GET response processing is measurably faster.
Contributed by Steve Loughran.
2024-01-17 18:34:14 +00:00
Hexiaoqiao
9634bd31e6
HADOOP-19031. Enhance access control for RunJar. ( #6427 ). Contributed by He Xiaoqiao.
...
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-01-17 15:00:06 +08:00
