big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,368 Commits 2 Branches 2 Tags 264 MiB
15b52fb6a4
Commit Graph

26368 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gautham B A
b1f418f802
YARN-11365. Fix NM class not found on Windows (#5093) 2022-10-31 09:19:07 -07:00
sabertiger
af7dd660e0
HADOOP-18233. Possible race condition with TemporaryAWSCredentialsProvider (#5024) 
This fixes a race condition with the TemporaryAWSCredentialProvider
one which has existed for a long time but which only surfaced
(usually in Spark) when the bucket existence probe was disabled
by setting fs.s3a.bucket.probe to 0 -a performance speedup
which was made the default in HADOOP-17454.

Contributed by Jimmy Wong.
 2022-10-31 12:43:30 +00:00
Ashutosh Gupta
cbe02c2e77
YARN-11264. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-tests (#4776) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:33:57 +09:00
Samrat
e04c9e810b
MAPREDUCE-7426. Fix typo in StartEndTimeBase (#4894) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 04:23:30 +09:00
Ashutosh Gupta
c096803387
YARN-11339. Upgrade Junit 4 to 5 in hadoop-yarn-services-api (#4995) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 03:07:16 +09:00
PJ Fanning
d6a65a4180
HADOOP-18472. Upgrade to snakeyaml 1.33 (#4958) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 02:30:41 +09:00
Ashutosh Gupta
2aae7ffe08
YARN-11337. Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo (#4993) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-30 02:25:53 +09:00
slfan1989
070a2d4880
YARN-11332. [Federation] Improve FederationClientInterceptor#ThreadPool thread pool configuration. (#4982) 2022-10-28 15:39:04 -07:00
slfan1989
b1cd88c598
YARN-11229. [Federation] Add checkUserAccessToQueue REST APIs for Router. (#4929) 2022-10-28 15:37:35 -07:00
Chris Nauroth
bfb84cd7f6
YARN-11360: Add number of decommissioning/shutdown nodes to YARN cluster metrics. (#5060) 2022-10-28 11:07:01 -07:00
jianghuazhu
88f7f5bc01
HDFS-16802.Print options when accessing ClientProtocol#rename2(). (#5013) 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
 2022-10-28 16:31:20 +08:00
M1eyu2018
8396caa484
HDFS-16716. Improve appendToFile command: support appending on file with new block (#4697) 
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-10-27 19:03:15 +08:00
huhaiyang
d26c35b228
HDFS-16817. Remove useless DataNode lock related configuration (#5072) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: ZanderXu <zanderxu@apache.org>
 2022-10-27 17:28:19 +08:00
Takanobu Asanuma
545a556883
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. (#5079) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
 2022-10-27 14:39:01 +09:00
slfan1989
ba77530ff4
YARN-11357. Fix FederationClientInterceptor#submitApplication Can't Update SubClusterId (#5055) 2022-10-26 16:42:22 -07:00
Bence Kosztolnik
562b693374 YARN-11356. Upgrade DataTables to 1.11.5 to fix CVEs. Contributed by Bence Kosztolnik. 2022-10-26 22:29:01 +02:00
Mehakmeet Singh
fba46aa5bb
HADOOP-18499. S3A to support HTTPS web proxies (#5051) 
The option "fs.s3a.proxy.ssl.enabled" controls
whether the s3a connects to a proxy over HTTP (default) or HTTPS.
Set to "true" to use HTTPS.

Contributed by Mehakmeet Singh
 2022-10-26 11:45:20 +01:00
Wang Yu
37bff63c0f
Refactor CallerContext's constructor to eliminate duplicate code (#5070) 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Zander Xu <zanderxu@apache.org>
 2022-10-26 06:40:31 +08:00
FuzzingTeam
f140506d67
HADOOP-18504. Fixed an unhandled NullPointerException in class KeyProvider (#5064) 
Contributed by FuzzingTeam
 2022-10-25 18:07:49 +01:00
Ashutosh Gupta
0a26d84df1
HADOOP-9946. NumAllSinks metrics shows lower value than NumActiveSinks (#5002) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:22:25 +08:00
Ashutosh Gupta
e6edbf1b4b
YARN-11338. Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher (#4994) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:21:12 +09:00
Ashutosh Gupta
21b7790866
YARN-11336. Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp (#4992) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 16:52:31 +09:00
slfan1989
454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
Gautham B A
833750f72a
HADOOP-18506. Update build instructions for Windows using VS2019 (#5066) 2022-10-24 09:28:29 -07:00
PJ Fanning
aac87ffe76
MAPREDUCE-7411: use secure XML parsers in mapreduce modules (#4980) 
Lockdown of parsers in hadoop-mapreduce.

Follow-on to HADOOP-18469. Add secure XML parser factories to XMLUtils

Contributed by P J Fanning
 2022-10-21 14:02:11 +01:00
slfan1989
d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
Willi Raschkowski
c4aa41aa80
HADOOP-18500. Upgrade maven-shade-plugin to 3.3.0 (#5045) 
Contributed by Willi Raschkowski
 2022-10-20 18:47:33 +01:00
FuzzingTeam
7f69e09290
HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in DefaultStringifier (#4957) 
Contributed by FuzzingTeam
 2022-10-20 18:12:17 +01:00
Sneha Vijayarajan
a996d889ec
HADOOP-17767. ABFS: Update test scripts (#3124) 
Contributed by Sneha Vijayarajan
 2022-10-20 18:07:04 +01:00
slfan1989
9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
jianghuazhu
c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. (#5031) 2022-10-20 10:58:23 +08:00
slfan1989
48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
Viraj Jasani
8aa04b0b24
HADOOP-18189 S3APrefetchingInputStream to support status probes when closed (#5036) 
Contributed by Viraj Jasani
 2022-10-19 14:38:11 +01:00
Daniel Carl Jones
6207ac47e0
HADOOP-18304. Improve user-facing S3A committers documentation (#4478) 
Contributed by: Daniel Carl Jones
 2022-10-19 12:56:47 +05:30
Steve Loughran
d80db6c9e5
HADOOP-18476. Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer (#4966) 
This is to try and close the underlying filesystems when the FileContext APIs are used.
Without this, threads may be leaked
 2022-10-18 14:53:02 +01:00
slfan1989
ee886cacd7
YARN-11247. Remove unused classes introduced by YARN-9615. (#4720) 2022-10-18 09:40:39 -04:00
Hexiaoqiao
babb050fa3
HADOOP-18497. Upgrade commons-text version to fix CVE-2022-42889. (#5037). Contributed by PJ Fanning. 
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-10-18 11:28:56 +08:00
Ankit Saurabh
2d91daab5e
HADOOP-18156. Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc (#4965) 
Contributed by Ankit Saurabh
 2022-10-17 18:10:47 +01:00
Ashutosh Gupta
9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (#4750) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-17 18:56:15 +08:00
ZanderXu
136291d2d5
HADOOP-18462. InstrumentedWriteLock should consider Reentrant case (#4919). Contributed by ZanderXu. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-17 12:44:25 +08:00
PJ Fanning
4ff6c9b8de
HADOOP-18493: upgrade jackson-databind to 2.12.7.1 (#5011). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-17 10:03:10 +05:30
Szilard Nemeth
b0d5182c31 YARN-10680. Revisit try blocks without catch blocks but having finally blocks. Contributed by Susheel Gupta 2022-10-15 21:51:08 +02:00
ahmarsuhail
77e551a478
HADOOP-18481. AWS v2 SDK upgrade log to not about standard AWS Credential Providers. (#4973) 
The AWS SDKV2 upgrade log no longer warns about instantiation
of the v1 SDK credential providers which are commonly used in
s3a configurations:

* com.amazonaws.auth.EnvironmentVariableCredentialsProvider
* com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper
* com.amazonaws.auth.InstanceProfileCredentialsProvider

When the hadoop-aws module moves to the v2 SDK, references to these
credential providers will be rewritten to their v2 equivalents.

Follow-on to HADOOP-18382. "Upgrade AWS SDK to V2 - Prerequisites"

Contributed by Ahmar Suhail
 2022-10-14 10:48:09 +01:00
slfan1989
5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989
1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989
647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
monthonk
9439d8e4e4
HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489) 
Follow-on from HADOOP-12020.

Contributed by Monthon Klongklaew
 2022-10-13 13:13:36 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30