Commit Graph

26743 Commits

Author SHA1 Message Date
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
huhaiyang
e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2023-01-11 09:12:26 +08:00
Chengbing Liu
4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246)
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
2023-01-10 10:03:25 -08:00
huhaiyang
f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Dongjoon Hyun
6f99558c2e
HADOOP-18590. Publish SBOM artifacts (#5281)
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
2023-01-09 16:41:06 -08:00
Simbarashe Dzinamarira
b56d483258
HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. (#5282) 2023-01-09 23:55:13 +00:00
ahmarsuhail
9c6eeb699e
HADOOP-18320. Fixes typos in Delegation Tokens documentation. (#4499)
Contributed By: Ahmar Suhail
2023-01-09 22:18:41 +05:30
Riya Khandelwal
dd49077aed
YARN-6412 aux-services classpath not documented (#5242) 2023-01-09 15:30:24 +01:00
Surendra Singh Lilhore
a65d24488a
HADOOP-18581 : Handle Server KDC re-login when Server and Client run … (#5248)
* HADOOP-18581 : Handle Server KDC re-login when Server and Client run in same JVM.
2023-01-08 23:55:06 +05:30
Simbarashe Dzinamarira
cd19da1309
HDFS-16877: Enables state context for namenode in TestObserverWithRouter (#5257) 2023-01-07 00:18:35 +00:00
PJ Fanning
b9eb760ed2
HADOOP-18587: upgrade to jettison 1.5.3 due to cve (#5270)
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
2023-01-06 15:35:50 -08:00
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
Yubi Lee
4511c360b9
HDFS-16883. Duplicate field name in hdfs-default.xml (#5271). Contributed by YUBI LEE. 2023-01-05 04:58:43 +05:30
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Ayush Saxena
b93b1c69cc
HADOOP-18586. Update the year to 2023. (#5265). Contributed by Ayush Saxena.
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2023-01-01 22:36:33 +05:30
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187)
Signed-off-by: Shilun Fan <slfan1989@apache.org>
2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao.
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
2022-12-27 15:58:53 -08:00
Neil
d25c1be517
HDFS-16861. RBF. Truncate API always fails when dirs use AllResolver oder on Router (#5184)
Co-authored-by: xiezhineng <xiezhineng@corp.netease.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-12-26 21:19:33 +08:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243)
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
2022-12-24 04:33:35 +09:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
2022-12-23 17:15:36 +01:00
ZanderXu
df093ef9af
HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time (#5098) 2022-12-23 14:16:42 +08:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231)
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
2022-12-22 17:20:53 +01:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744)
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-12-21 10:06:01 -08:00
David Dillon
b63b777c84
HDFS-16873 FileStatus compareTo specify ordering by path (#5219) 2022-12-21 10:11:55 +08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872)
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-12-20 15:50:58 -08:00
Daniel-009497
7ff326129d
HDFS-16871 DiskBalancer process may throw IllegalArgumentException when the target DataNode has capital letter in hostname (#5240) 2022-12-21 07:12:02 +08:00
陈爽-Jack Chen
f6605f1b3a
HADOOP-18438: AliyunOSSFileSystemStore deleteObjects interface should return the objects that failed to delete (#4857)
Merged to trunk, thank @chenshuang778  for your contribution
2022-12-20 13:57:49 +08:00
Steve Loughran
52c72fafe4
HADOOP-18470. Update index md with section on ABFS prefetching 2022-12-19 13:04:26 +00:00
PJ Fanning
6a07b5dc10
HADOOP-18575. Make XML transformer factory more lenient (#5224)
Due diligence followup to
HADOOP-18469. Add secure XML parser factories to XMLUtils (#4940)

Contributed by P J Fanning
2022-12-18 12:25:10 +00:00
Steve Loughran
33785fc5ad
HADOOP-18577. Followup: javadoc fix (#5232)
Fixes a javadoc error which came with
HADOOP-18577. ABFS: Add probes of readahead fix (#5205)

Part of the HADOOP-18521 ABFS readahead fix; MUST be included.

Contributed by Steve Loughran
2022-12-18 12:19:33 +00:00
Chengbing Liu
ca3526da92
HADOOP-18567. LogThrottlingHelper: properly trigger dependent recorders in cases of infrequent logging (#5215)
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
2022-12-16 09:15:11 -08:00
Xing Lin
f7bdf6c667
HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down (#5160)
Signed-off-by: Erik Krogen <xkrogen@apache.org>
2022-12-16 08:46:14 -08:00
Happy-shi
c5b42d59d2
HDFS-16866. Fix a typo in Dispatcher (#5202)
Signed-off-by: Tao Li <tomscut@apache.org>
2022-12-16 11:07:41 +08:00
Steve Loughran
cf1244492d
HADOOP-18577. ABFS: Add probes of readahead fix (#5205)
Followup patch to  HADOOP-18456 as part of HADOOP-18521,
ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

Add probes of readahead fix aid in checking safety of
hadoop ABFS client across different releases.

* ReadBufferManager constructor logs the fact it is safe at TRACE
* AbfsInputStream declares it is fixed in toString()
  by including fs.azure.capability.readahead.safe" in the
  result.

The ABFS FileSystem hasPathCapability("fs.azure.capability.readahead.safe")
probe returns true to indicate the client's readahead manager has been fixed
to be safe when prefetching.

All Hadoop releases for which probe this returns false
and for which the probe "fs.capability.etags.available"
returns true at risk of returning invalid data when reading
ADLS Gen2/Azure storage data.

Contributed by Steve Loughran.
2022-12-15 17:08:25 +00:00
Steve Loughran
5f08e51b72
HADOOP-18561. Update commons-net to 3.9.0 (#5214)
Addresses CVE-2021-37533, which *only* relates to FTP.

Applications not using the ftp:// filesystem, which, as
anyone who has used it will know is very minimal and
so rarely used, is not a critical part of the project.

Furthermore, the FTP-related issue is at worst information leakage
if someone connects to a malicious server.

This is a due diligence PR rather than an emergency fix.

Contributed by Steve Loughran
2022-12-15 16:45:05 +00:00
Steve Loughran
f7b1bb4dcc
HADOOP-18573. Improve error reporting on non-standard kerberos names (#5221)
The kerberos RPC does not declare any restriction on
characters used in kerberos names, though
implementations MAY be more restrictive.

If the kerberos controller supports use non-conventional
principal names *and the kerberos admin chooses to use them*
this can confuse some of the parsing.

The obvious solution is for the enterprise admins to "not do that"
as a lot of things break, bits of hadoop included.

Harden the hadoop code slightly so at least we fail more gracefully,
so people can then get in touch with their sysadmin and tell them
to stop it.
2022-12-15 11:42:36 +00:00
Mehakmeet Singh
32414cfe46
HADOOP-18574. Changing log level of IOStatistics increment to make the DEBUG logs less noisy (#5223)
Contributed by: Mehakmeet Singh
2022-12-15 10:19:18 +05:30
slfan1989
6172c3192d
YARN-11358. [Federation] Add FederationInterceptor#allow-partial-result config. (#5056) 2022-12-14 14:37:56 -08:00
Steve Loughran
aaf92fe183
HADOOP-18526. Leak of S3AInstrumentation instances via hadoop Metrics references (#5144)
This has triggered an OOM in a process which was churning through s3a fs
instances; the increased memory footprint of IOStatistics amplified what
must have been a long-standing issue with FS instances being created
and not closed()

*  Makes sure instrumentation is closed when the FS is closed.
*  Uses a weak reference from metrics to instrumentation, so even
   if the FS wasn't closed (see HADOOP-18478), this back reference
   would not cause the S3AInstrumentation reference to be retained.
*  If S3AFileSystem is configured to log at TRACE it will log the
   calling stack of initialize(), so help identify where the
   instance is being created. This should help track down
   the cause of instance leakage.

Contributed by Steve Loughran.
2022-12-14 18:21:03 +00:00
slfan1989
63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. (#5131) 2022-12-14 09:09:38 -08:00
Doroszlai, Attila
4de8791deb
HADOOP-18569. NFS Gateway may release buffer too early (#5212)
(cherry picked from commit df4812df65)
2022-12-14 15:55:44 +01:00
Steve Loughran
1cecf8ab70
HADOOP-18183. s3a audit logs to publish range start/end of GET requests. (#5110)
The start and end of the range is set in a new audit param "rg",
e.g "?rg=100-200"

Contributed by Ankit Saurabh
2022-12-14 14:01:28 +00:00
Ashutosh Gupta
85ec7969a7
MAPREDUCE-7428. Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp (#5209)
Contributed by: Ashutosh Gupta
2022-12-14 12:54:08 +00:00
curie71
fdcbc8b072
HDFS-16868. Fix audit log duplicate issue when an ACE occurs in FSNamesystem. (#5206). Contributed by Beibei Zhao.
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-12-13 12:24:51 +08:00
slfan1989
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. (#5182) 2022-12-10 15:03:49 -08:00
Jack Richard Buggins
a46b20d25f
HADOOP-18329. Support for IBM Semeru JVM > 11.0.15.0 Vendor Name Changes (#4537)
The static boolean PlatformName.IBM_JAVA now identifies
Java 11+ IBM Semeru runtimes as IBM JVM releases.

Contributed by Jack Buggins.
2022-12-10 14:27:05 +00:00
Steve Loughran
0a7dfcc332
HADOOP-18546. Followup: ITestReadBufferManager fix (#5198)
This is a followup to the original HADOOP-18546
patch; cherry-picks of that should include this
or follow up with it.

Removes risk of race conditions in assertions
of ITestReadBufferManager on the state of the in-progress
and completed queues by removing assertions brittle
to race conditions in scheduling/network IO

* Waits for all the executor pool shutdown to complete before
  making any assertions
* Assertions that there are no in progress reads MUST be
  cut as there may be some and they won't be cancelled.
* Assertions that the completed list is without buffers
  of a closed stream are brittle because if there was
  an in progress stream which completed after stream.close()
  then it will end up in the list.

Contributed by Steve Loughran
2022-12-09 13:47:11 +00:00