big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,454 Commits 2 Branches 2 Tags 264 MiB
e2ab35084a
Commit Graph

26454 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nikita Eshkeev
4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
PJ Fanning
d81d98388c
HADOOP-18575: followup: try to avoid repeatedly hitting exceptions when transformer factories do not support attributes (#5253) 
Part of HADOOP-18469 and the hardening of XML/XSL parsers.
Followup to the main HADOOP-18575 patch, to improve performance when
working with xml/xsl engines which don't support the relevant attributes.

Include this change when backporting.

Contributed by PJ Fanning.
 2023-01-16 13:15:37 +00:00
Ashutosh Gupta
38453f8589
MAPREDUCE-7413. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-hs-plugins (#5023) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2023-01-16 18:19:28 +09:00
Ashutosh Gupta
082266516a
MAPREDUCE-7417. Upgrade Junit 4 to 5 in hadoop-mapreduce-client-uploader (#5019) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2023-01-16 17:22:04 +09:00
slfan1989
168fa07801
YARN-11409. Fix Typo of ResourceManager#webapp moudle. (#5285) 2023-01-13 11:53:13 +08:00
slfan1989
4520448327
YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. (#5193) 2023-01-12 15:04:09 -08:00
Viraj Jasani
1263e024b9
HDFS-16887 Log start and end of phase/step in startup progress (#5292) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 14:26:52 -08:00
skysiders
36bf54aba0
MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs (#4237) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 13:48:29 -08:00
huangxiaoping
a90e424d9f
HADOOP-18591. Fix a typo in Trash (#5291) 
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-12 13:21:21 -08:00
slfan1989
3d21cff263
YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. (#5289) 
* YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

* YARN-11413. Fix CheckStyle.

* YARN-11413. Fix CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
 2023-01-12 14:29:05 +01:00
Simbarashe Dzinamarira
f26d8bc9bd
HDFS-16886: Fixes error in documentation for StateStoreRecordsOperations. (#5290) 2023-01-11 19:46:51 +00:00
Szilard Nemeth
7f6cc196f8
YARN-11190. CS Mapping rule bug: User matcher does not work correctly for usernames with dot (#4471) 2023-01-11 13:23:04 +01:00
huhaiyang
e3b09b7512
HDFS-16884. Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed (#5280) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-01-11 09:12:26 +08:00
Chengbing Liu
4cf304de45
HDFS-16872. Fix log throttling by declaring LogThrottlingHelper as static members (#5246) 
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-01-10 10:03:25 -08:00
huhaiyang
f3cff032e6
HDFS-16885. Fix TestHdfsConfigFields#testCompareConfigurationClassAgainstXml failed (#5283) 2023-01-10 09:02:56 +08:00
Dongjoon Hyun
6f99558c2e
HADOOP-18590. Publish SBOM artifacts (#5281) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-09 16:41:06 -08:00
Simbarashe Dzinamarira
b56d483258
HDFS-16876: Changes cleanup of shared RouterStateIdContext to be driven by namenodeResolver data. (#5282) 2023-01-09 23:55:13 +00:00
ahmarsuhail
9c6eeb699e
HADOOP-18320. Fixes typos in Delegation Tokens documentation. (#4499) 
Contributed By: Ahmar Suhail
 2023-01-09 22:18:41 +05:30
Riya Khandelwal
dd49077aed
YARN-6412 aux-services classpath not documented (#5242) 2023-01-09 15:30:24 +01:00
Surendra Singh Lilhore
a65d24488a
HADOOP-18581 : Handle Server KDC re-login when Server and Client run … (#5248) 
* HADOOP-18581 : Handle Server KDC re-login when Server and Client run in same JVM.
 2023-01-08 23:55:06 +05:30
Simbarashe Dzinamarira
cd19da1309
HDFS-16877: Enables state context for namenode in TestObserverWithRouter (#5257) 2023-01-07 00:18:35 +00:00
PJ Fanning
b9eb760ed2
HADOOP-18587: upgrade to jettison 1.5.3 due to cve (#5270) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-06 15:35:50 -08:00
Tsz-Wo Nicholas Sze
5022003e0f
HDFS-16881. Warn if AccessControlEnforcer runs for a long time to check permission. (#5268) 2023-01-05 09:31:52 +08:00
Yubi Lee
4511c360b9
HDFS-16883. Duplicate field name in hdfs-default.xml (#5271). Contributed by YUBI LEE. 2023-01-05 04:58:43 +05:30
huhaiyang
35ce60eadd
HDFS-16879. EC: Fsck -blockId shows number of redundant internal block replicas for EC Blocks (#5264) 2023-01-04 11:38:32 +08:00
slfan1989
0926fa5a2c
YARN-11225. [Federation] Add postDelegationToken postDelegationTokenExpiration cancelDelegationToken REST APIs for Router. (#5185) 2023-01-03 02:14:02 -08:00
susheel-gupta
c44c9f984b
YARN-11393. Fs2cs could be extended to set ULF to -1 upon conversion (#5201) 2023-01-02 15:35:16 +01:00
Ayush Saxena
b93b1c69cc
HADOOP-18586. Update the year to 2023. (#5265). Contributed by Ayush Saxena. 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-01 22:36:33 +05:30
Chris Nauroth
6b67373d10
YARN-11388: Prevent resource leaks in TestClientRMService. (#5187) 
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2022-12-28 11:00:27 -08:00
curie71
9668a85d40
YARN-11392 Audit Log missing in ClientRMService (#5250). Contributed by Beibei Zhao. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-12-27 15:58:53 -08:00
Neil
d25c1be517
HDFS-16861. RBF. Truncate API always fails when dirs use AllResolver oder on Router (#5184) 
Co-authored-by: xiezhineng <xiezhineng@corp.netease.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-26 21:19:33 +08:00
Akira Ajisaka
049d1762bd
MAPREDUCE-7428. Fix failing MapReduce tests due to the JUnit upgrades in WebServicesTestUtils (#5243) 
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
 2022-12-24 04:33:35 +09:00
Bence Kosztolnik
bf8ab83cd0 YARN-11395. RM UI, RMAttemptBlock can not render FINAL_SAVING. Contributed by Bence Kosztolnik 
- In the YARN-1345 remove of FINAL_SAVING was missed from RMAttemptBlock
- Same issue was present after YARN-1345 in YARN-4411
- YARN-4411 logic was applied in this commit for FINAL_SAVING
 2022-12-23 17:15:36 +01:00
ZanderXu
df093ef9af
HDFS-16831. [RBF SBN] GetNamenodesForNameserviceId should shuffle Observer NameNodes every time (#5098) 2022-12-23 14:16:42 +08:00
slfan1989
17035da46e
YARN-11226. [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router. (#5175) 2022-12-22 11:25:09 -08:00
susheel-gupta
e6056d128a
YARN-10879. Incorrect WARN text in ACL check for application tag based placement (#5231) 
Change-Id: Id892e38fe4c834b1743a0df2f0a40146d3d5a878
 2022-12-22 17:20:53 +01:00
ZanderXu
15b52fb6a4
HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer (#4744) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-21 10:06:01 -08:00
David Dillon
b63b777c84
HDFS-16873 FileStatus compareTo specify ordering by path (#5219) 2022-12-21 10:11:55 +08:00
ZanderXu
8d221255f2
HDFS-16764. [SBN Read] ObserverNamenode should throw ObserverRetryOnActiveException instead of FileNotFoundException during processing of addBlock rpc (#4872) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-12-20 15:50:58 -08:00
Daniel-009497
7ff326129d
HDFS-16871 DiskBalancer process may throw IllegalArgumentException when the target DataNode has capital letter in hostname (#5240) 2022-12-21 07:12:02 +08:00
陈爽-Jack Chen
f6605f1b3a
HADOOP-18438: AliyunOSSFileSystemStore deleteObjects interface should return the objects that failed to delete (#4857) 
Merged to trunk, thank @chenshuang778  for your contribution
 2022-12-20 13:57:49 +08:00
Steve Loughran
52c72fafe4
HADOOP-18470. Update index md with section on ABFS prefetching 2022-12-19 13:04:26 +00:00
PJ Fanning
6a07b5dc10
HADOOP-18575. Make XML transformer factory more lenient (#5224) 
Due diligence followup to
HADOOP-18469. Add secure XML parser factories to XMLUtils (#4940)

Contributed by P J Fanning
 2022-12-18 12:25:10 +00:00
Steve Loughran
33785fc5ad
HADOOP-18577. Followup: javadoc fix (#5232) 
Fixes a javadoc error which came with
HADOOP-18577. ABFS: Add probes of readahead fix (#5205)

Part of the HADOOP-18521 ABFS readahead fix; MUST be included.

Contributed by Steve Loughran
 2022-12-18 12:19:33 +00:00
Chengbing Liu
ca3526da92
HADOOP-18567. LogThrottlingHelper: properly trigger dependent recorders in cases of infrequent logging (#5215) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: Chengbing Liu <liuchengbing@qiyi.com>
 2022-12-16 09:15:11 -08:00
Xing Lin
f7bdf6c667
HDFS-16852. Skip KeyProviderCache shutdown hook registration if already shutting down (#5160) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-12-16 08:46:14 -08:00
Happy-shi
c5b42d59d2
HDFS-16866. Fix a typo in Dispatcher (#5202) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-12-16 11:07:41 +08:00
Steve Loughran
cf1244492d
HADOOP-18577. ABFS: Add probes of readahead fix (#5205) 
Followup patch to  HADOOP-18456 as part of HADOOP-18521,
ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

Add probes of readahead fix aid in checking safety of
hadoop ABFS client across different releases.

* ReadBufferManager constructor logs the fact it is safe at TRACE
* AbfsInputStream declares it is fixed in toString()
  by including fs.azure.capability.readahead.safe" in the
  result.

The ABFS FileSystem hasPathCapability("fs.azure.capability.readahead.safe")
probe returns true to indicate the client's readahead manager has been fixed
to be safe when prefetching.

All Hadoop releases for which probe this returns false
and for which the probe "fs.capability.etags.available"
returns true at risk of returning invalid data when reading
ADLS Gen2/Azure storage data.

Contributed by Steve Loughran.
 2022-12-15 17:08:25 +00:00
Steve Loughran
5f08e51b72
HADOOP-18561. Update commons-net to 3.9.0 (#5214) 
Addresses CVE-2021-37533, which *only* relates to FTP.

Applications not using the ftp:// filesystem, which, as
anyone who has used it will know is very minimal and
so rarely used, is not a critical part of the project.

Furthermore, the FTP-related issue is at worst information leakage
if someone connects to a malicious server.

This is a due diligence PR rather than an emergency fix.

Contributed by Steve Loughran
 2022-12-15 16:45:05 +00:00
Steve Loughran
f7b1bb4dcc
HADOOP-18573. Improve error reporting on non-standard kerberos names (#5221) 
The kerberos RPC does not declare any restriction on
characters used in kerberos names, though
implementations MAY be more restrictive.

If the kerberos controller supports use non-conventional
principal names *and the kerberos admin chooses to use them*
this can confuse some of the parsing.

The obvious solution is for the enterprise admins to "not do that"
as a lot of things break, bits of hadoop included.

Harden the hadoop code slightly so at least we fail more gracefully,
so people can then get in touch with their sysadmin and tell them
to stop it.
 2022-12-15 11:42:36 +00:00