Commit Graph

27139 Commits

Author SHA1 Message Date
huhaiyang
6464507cd1
HDFS-17369. Add uuid into datanode info for NameNodeMXBean (#6521) Contributed by Haiyang Hu.
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-02-03 19:20:04 +08:00
Xing Lin
d74e5160cd
HADOOP-19061 Capture exception from rpcRequestSender.start() in IPC.Connection.run() (#6519)
* HADOOP-19061 - Capture exception from rpcRequestSender.start() in IPC.Connection.run() and proper cleaning is followed if an exception is thrown.

---------

Co-authored-by: Xing Lin <xinglin@linkedin.com>
2024-02-02 16:22:16 -08:00
Viraj Jasani
d278b349f6
HADOOP-19044. S3A: AWS SDK V2 - Update region logic (#6479)
Improves region handling in the S3A connector, including enabling cross-region support
when that is considered necessary.

Consult the documentation in connecting.md/connecting.html for the current
resolution process.

Contributed by Viraj Jasani
2024-02-02 17:07:05 +00:00
Viraj Jasani
7504b8505f
HADOOP-18980. S3A credential provider remapping: make extensible (#6406)
Contributed by Viraj Jasani
2024-02-02 17:02:48 +00:00
DieterDP
be13e94843
HADOOP-18987. Various fixes to FileSystem API docs (#6292)
Contributed by Dieter De Paepe
2024-02-02 11:49:31 +00:00
hfutatzhanghb
4f4b846986
HDFS-17359. EC: recheck failed streamers should only after flushing all packets. (#6503). Contributed by farmmamba.
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2024-02-01 23:53:37 +09:00
Takanobu Asanuma
2f1718c363
HADOOP-19056. Highlight RBF features and improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-31 13:30:35 +08:00
Yang Jiandan
98259fb4da
YARN-11653. Add Totoal_Memory and Total_Vcores columns in Nodes page (#6501) Contributed by Jiandan Yang.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-31 07:07:54 +08:00
Steve Loughran
8261229daa
HADOOP-18830. Cut S3 Select (#6144)
Cut out S3 Select
* leave public/unstable constants alone
* s3guard tool will fail with error
* s3afs. path capability will fail
* openFile() will fail with specific error
* s3 select doc updated
* Cut eventstream jar
* New test: ITestSelectUnsupported verifies new failure
  handling above

Contributed by Steve Loughran
2024-01-30 16:12:27 +00:00
Steve Loughran
6da1a19a83
HADOOP-19045. S3A: Validate CreateSession Timeout Propagation (#6470)
New test ITestCreateSessionTimeout to verify that the duration set
in fs.s3a.connection.request.timeout is passed all the way down.

This is done by adding a sleep() in a custom signer and verifying
that it is interrupted and that an AWSApiCallTimeoutException is
raised.

+ Fix testRequestTimeout()
* doesn't skip if considered cross-region
* sets a minimum duration of 0 before invocation
* resets the minimum afterwards

Contributed by Steve Loughran
2024-01-30 15:32:24 +00:00
Yang Jiandan
dffb505725
YARN-11650. Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue (#6490) Contributed by Jiandan Yang.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-29 21:37:32 +08:00
Peter Szucs
4046751939
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#6499) 2024-01-26 14:42:59 +01:00
Benjamin Teke
897f446d54
HADOOP-19051: Highlight Capacity Scheduler new features in release for the release 3.4.0 (#6500) Contributed by Benjamin Teke.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-26 13:33:55 +08:00
Tsz-Wo Nicholas Sze
da34ecdb83
HADOOP-19035. CrcUtil/CrcComposer should not throw IOException for non-IO. (#6443) 2024-01-25 10:35:32 -08:00
Lei313
ac471d7daa
HDFS-17339:Skip cacheReport when one blockPool does not have CacheBlock on this DataNode (#6456). Contributed by lei w.
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 21:15:20 +08:00
Tamas Domok
7272a3d960
YARN-11641. Fix getCapacityConfigurationTypeForQueues for absolute mode with zero resource. (#6435) 2024-01-25 13:58:00 +01:00
slfan1989
38f10c657e
HADOOP-19039. Hadoop 3.4.0 Highlight big features and improvements. (#6462) Contributed by Shilun Fan.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-25 15:42:21 +08:00
huhaiyang
caba9bbab3
HDFS-17346. Fix DirectoryScanner check mark the normal blocks as corrupt (#6476). Contributed by Haiyang Hu.
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-25 12:49:18 +08:00
slfan1989
6b80b1e60f
Revert "YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup" (#6497) 2024-01-25 11:44:15 +08:00
Peter Szucs
dd72e8fbc3
YARN-11041. Replace all occurences of queuePath with the new QueuePath class - followup (#5332) 2024-01-24 15:51:07 +01:00
Tamas Domok
af5c8e70ea
YARN-11645. Order the queues by queuePath in json assert tests. (#6432) 2024-01-24 10:12:24 +01:00
LiuGuH
8193a84020
HDFS-17324. RBF: Router should not return nameservices that not enable observer r… (#6412) 2024-01-23 12:30:08 -08:00
jialiang
b2fac14828
HADOOP-19019: Parallel Maven Build Support for Apache Hadoop (#6373). Contributed by JiaLiangC.
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-01-23 14:51:20 +08:00
Ferenc Erdelyi
8243da8cb0
YARN-11639. CME and NPE in PriorityUtilizationQueueOrderingPolicy (#6455) 2024-01-22 15:41:48 +01:00
hfutatzhanghb
54f7a6b127
HDFS-17293. First packet data + checksum size will be set to 516 bytes when writing to a new block. (#6368). Contributed by farmmamba.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-01-22 11:50:51 +08:00
Pranav Saxena
7dc166ddc7
HADOOP-18883. [ABFS]: Expect-100 JDK bug resolution: prevent multiple server calls (#6022)
Address JDK bug JDK-8314978 related to handling of HTTP 100
responses. 

https://bugs.openjdk.org/browse/JDK-8314978

In the AbfsHttpOperation, after sendRequest() we call processResponse()
method from AbfsRestOperation.
Even if the conn.getOutputStream() fails due to expect-100 error, 
we consume the exception and let the code go ahead.
This may call getHeaderField() / getHeaderFields() / getHeaderFieldLong() after
getOutputStream() has failed. These invocation all lead to server calls.

This commit aims to prevent this.
If connection.getOutputStream() fails due to an Expect-100 error,
the ABFS client does not invoke getHeaderField(), getHeaderFields(),
getHeaderFieldLong() or getInputStream().

getResponseCode() is safe as on the failure it sets the
responseCode variable in HttpUrlConnection object.

Contributed by Pranav Saxena
2024-01-21 19:14:54 +00:00
Steve Loughran
d274f778c1
HADOOP-19046. S3A: update AWS V2 SDK to 2.23.5; v1 to 1.12.599 (#6467)
This update ensures that the timeout set in fs.s3a.connection.request.timeout is passed down
to calls to CreateSession made in the AWS SDK to get S3 Express session tokens.

Contributed by Steve Loughran
2024-01-21 19:00:34 +00:00
PJ Fanning
76691dfa14
HADOOP-18894: upgrade sshd-core due to CVEs (#6060) Contributed by PJ Fanning.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-21 08:13:25 +08:00
LiuGuH
2a1ee8dfcd
HDFS-17311. RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue. (#6392) Contributed by liuguanghua.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:55:23 +08:00
slfan1989
15e1789baf
Revert "HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998)" (#6457) Contributed by Shilun Fan.
This reverts commit c1bf3cb0.

Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:51:55 +08:00
Susheel Gupta
d0df0689b4
YARN-11607: TestTimelineAuthFilterForV2 fails intermittently (#6459) Contributed by Susheel Gupta.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-20 07:42:08 +08:00
Jian Zhang
1036544480
HDFS-17302. RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation. (#6380) 2024-01-19 14:02:21 -08:00
slfan1989
8444f69511
Preparing for 3.5.0 development (#6411)
Co-authored-by: slfan1989 <slfan1989@apache.org>
2024-01-19 15:05:22 +08:00
Xing Lin
27ecc23ae7
HDFS-17332 DFSInputStream: avoid logging stacktrace until when we really need to fail a read request with a MissingBlockException (#6446)
Print a warn log message for read retries and only print the full stack trace for a read request failure.

Contributed by: Xing Lin
2024-01-18 18:03:28 -08:00
Lei313
cc4c4be1b7
HDFS-17331:Fix Blocks are always -1 and DataNode version are always UNKNOWN in federationhealth.html (#6429). Contributed by lei w.
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
2024-01-18 21:10:54 +08:00
slfan1989
4c3d4e6a57
HADOOP-19038. Improve create-release RUN script. (#6448) Contributed by Shilun Fan.
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-18 19:12:12 +08:00
PJ Fanning
04e447cfa7
YARN-11647. use StandardCharsets.UTF_8 (#6447) Contributed by PJ Fanning.
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-18 13:53:18 +08:00
hfutatzhanghb
ba6ada73ac
HDFS-17337. RPC RESPONSE time seems not exactly accurate when using FSEditLogAsync. (#6439). Contributed by farmmamba.
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-01-18 11:10:05 +08:00
Steve Loughran
eeb657e85f
HADOOP-19033. S3A: disable checksums when fs.s3a.checksum.validation = false (#6441)
Add new option fs.s3a.checksum.validation, default false, which
is used when creating s3 clients to enable/disable checksum
validation.

When false, GET response processing is measurably faster.

Contributed by Steve Loughran.
2024-01-17 18:34:14 +00:00
Hexiaoqiao
9634bd31e6
HADOOP-19031. Enhance access control for RunJar. (#6427). Contributed by He Xiaoqiao.
Signed-off-by: Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2024-01-17 15:00:06 +08:00
Mukund Thakur
7b1570e2f1
HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)
HADOOP-19015.  Increase fs.s3a.connection.maximum to 500 to minimize the risk of Timeout waiting for connection from the pool

Contributed By: Mukund Thakur
2024-01-16 17:06:28 -06:00
Steve Loughran
d378853790
HADOOP-18975 S3A: Add option fs.s3a.endpoint.fips to use AWS FIPS endpoints (#6277)
Adds a new option `fs.s3a.endpoint.fips` to switch the SDK client to use
FIPS endpoints, as an alternative to explicitly declaring them.


* The option is available as a path capability for probes.
* SDK v2 itself doesn't know that some regions don't have FIPS endpoints
* SDK only fails with endpoint + fips flag as a retried exception; wit this
  change the S3A client should fail fast.
  PR fails fast.
* Adds a new "connecting.md" doc; moves existing docs there and restructures.
* New Tests in ITestS3AEndpointRegion

bucket-info command support:

* added to list of path capabilities
* added -fips flag and test for explicit probe
* also now prints bucket region
* and removed some of the obsolete s3guard options
* updated docs

Contributed by Steve Loughran
2024-01-16 14:16:12 +00:00
Steve Loughran
36198b5edf
HADOOP-19027. S3A: S3AInputStream doesn't recover from HTTP/channel exceptions (#6425)
Differentiate from "EOF out of range/end of GET" from
"EOF channel problems" through
two different subclasses of EOFException and input streams to always
retry on http channel errors; out of range GET requests are not retried.
Currently an EOFException is always treated as a fail-fast call in read()

This allows for all existing external code catching EOFException to handle
both, but S3AInputStream to cleanly differentiate range errors (map to -1)
from channel errors (retry)

- HttpChannelEOFException is subclass of EOFException, so all code
  which catches EOFException is still happy.
  retry policy: connectivityFailure
- RangeNotSatisfiableEOFException is the subclass of EOFException
  raised on 416 GET range errors.
  retry policy: fail
- Method ErrorTranslation.maybeExtractChannelException() to create this
  from shaded/unshaded NoHttpResponseException, using string match to
  avoid classpath problems.
- And do this for SdkClientExceptions with OpenSSL error code WFOPENSSL0035.
  We believe this is the OpenSSL equivalent.
- ErrorTranslation.maybeExtractIOException() to perform this translation as
  appropriate.

S3AInputStream.reopen() code retries on EOF, except on
 RangeNotSatisfiableEOFException,
 which is converted to a -1 response to the caller
 as is done historically.

S3AInputStream knows to handle these with
 read(): HttpChannelEOFException: stream aborting close then retry
 lazySeek(): Map RangeNotSatisfiableEOFException to -1, but do not map
  any other EOFException class raised.

This means that
* out of range reads map to -1
* channel problems in reopen are retried
* channel problems in read() abort the failed http connection so it
  isn't recycled

Tests for this using/abusing mocking.

Testing through actually raising 416 exceptions and verifying that
readFully(), char read() and vector reads are all good.

There is no attempt to recover within a readFully(); there's
a boolean constant switch to turn this on, but if anyone does
it a test will spin forever as the inner PositionedReadable.read(position, buffer, len)
downgrades all EOF exceptions to -1.
A new method would need to be added which controls whether to downgrade/rethrow
exceptions.

What does that mean? Possibly reduced resilience to non-retried failures
on the inner stream, even though more channel exceptions are retried on.

Contributed by Steve Loughran
2024-01-16 14:14:03 +00:00
slfan1989
6652922333
HADOOP-19040. mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes. (#6450) Contributed by Shilun Fan.
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-16 22:11:16 +08:00
slfan1989
827e33601e
YARN-11638. [GPG] GPG Support CLI. (#6396) Contributed by Shilun Fan.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2024-01-16 21:49:51 +08:00
Benjamin Teke
f6fea5da2a
MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests. (#6451) 2024-01-15 14:24:56 +01:00
slfan1989
6ebce65ae8
YARN-11634. [Addendum] Speed-up TestTimelineClient. (#6419)
Co-authored-by: slfan1989 <slfan1989@apache.org>
2024-01-15 08:44:17 +01:00
slfan1989
0f8b74b03f
HADOOP-19034. Fix Download Maven Url Not Found. (#6438). Contributed by Shilun Fan.
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2024-01-14 18:30:40 +08:00
hfutatzhanghb
a30681077b
HDFS-17291. DataNode metric bytesWritten is not totally accurate in some situations. (#6360). Contributed by farmmamba.
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-01-13 20:45:00 +08:00
hfutatzhanghb
ead7b7f565
HDFS-17289. Considering the size of non-lastBlocks equals to complete block size can cause append failure. (#6357). Contributed by farmmamba.
Reviewed-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: huangzhaobo <huangzhaobo99@126.com>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
2024-01-13 20:34:02 +08:00