big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,272 Commits 2 Branches 2 Tags 264 MiB
388f2f182f
Commit Graph

26272 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
PJ Fanning
bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
monthonk
9439d8e4e4
HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489) 
Follow-on from HADOOP-12020.

Contributed by Monthon Klongklaew
 2022-10-13 13:13:36 +01:00
slfan1989
3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989
1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
Ashutosh Gupta
d6b1e1eeb6
HDFS-16777. datatables@1.10.17 sonatype-2020-0988 vulnerability (#5003) 2022-10-12 14:51:12 -07:00
Gautham B A
5694d7e25f
Add Dockerfile_windows_10 (#4936) 2022-10-12 09:30:54 -07:00
slfan1989
d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989
9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989
82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Gautham B A
2122733c30
Add .yetus/excludes.txt (#4984) 2022-10-11 09:23:34 -07:00
belugabehr
03d600fa82
HADOOP-17779: Lock File System Creator Semaphore Uninterruptibly (#3158) 2022-10-11 11:56:32 +01:00
huhaiyang
d14b88c698
HDFS-16774.Improve async delete replica on datanode (#4903) 
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException

Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-11 16:22:56 +08:00
PJ Fanning
4fe079f85f
HDFS-16795. Use secure XML parsers (#4979) 
Contributed by P J Fanning
 2022-10-10 18:56:35 +01:00
Szilard Nemeth
0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
ZanderXu
62ff4e36cf
HDFS-16787. Remove the redundant lock in DataSetLockManager#removeLock (#4948). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 20:05:02 +08:00
ZanderXu
b0b2cb4a16
HDFS-16783. Remove the redundant lock in deepCopyReplica and getFinalizedBlocks (#4942). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 19:52:22 +08:00
Mukund Thakur
be70bbb4be
HADOOP-18460. checkIfVectoredIOStopped before populating the buffers (#4986) 
Contributed by Mukund Thakur
 2022-10-10 11:17:45 +01:00
Steve Loughran
540a660429
HADOOP-18480. Upgrade aws sdk to 1.12.316 (#4972) 
Contributed by Steve Loughran
 2022-10-10 10:23:50 +01:00
Ashutosh Gupta
9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
ZanderXu
b0bfd09c41
HDFS-16798. HDFS-16798. SerialNumberMap should decrease current counter if the item exist. (#4987). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-09 18:58:09 +08:00
PJ Fanning
5eddec8c46
HADOOP-18468: Upgrade jettison to 1.5.1 to fix CVE-2022-40149 (#4937) 
Contributed by PJ Fanning
 2022-10-07 15:44:01 +01:00
Ashutosh Gupta
062c50db6b
MAPREDUCE-7370. Parallelize MultipleOutputs#close call (#4248). Contributed by Ashutosh Gupta. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2022-10-06 15:23:05 -07:00
PJ Fanning
8336b91329
HADOOP-18469. Add secure XML parser factories to XMLUtils (#4940) 
Add to XMLUtils a set of methods to create secure XML Parsers/transformers, locking down DTD, schema, XXE exposure.

Use these wherever XML parsers are created.

Contributed by PJ Fanning
 2022-10-06 19:30:51 +01:00
slfan1989
b31b3ea0f6
YARN-11187. Remove WhiteBox in yarn module. (#4463) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-06 23:13:33 +09:00
Daniel Carl Jones
7ec762a5fd
HADOOP-18465. Fix S3A SSE test skip when encryption is disabled (#4925) 
Contributed by Daniel Carl Jones
 2022-10-06 12:42:01 +01:00
Alessandro Passaro
1675a28e5a
HADOOP-18378. Implement lazy seek in S3A prefetching. (#4955) 
Make S3APrefetchingInputStream.seek() completely lazy. Calls to seek() will not affect the current buffer nor interfere with prefetching, until read() is called.

This change allows various usage patterns to benefit from prefetching, e.g. when calling readFully(position, buffer) in a loop for contiguous positions the intermediate internal calls to seek() will be noops and prefetching will have the same performance as in a sequential read.

Contributed by Alessandro Passaro.
 2022-10-06 12:00:41 +01:00
Steve Loughran
38b2ed2151
HADOOP-18442. Remove openstack support (#4855) 
Contributed by Steve Loughran
 2022-10-06 11:49:38 +01:00
slfan1989
1a9faf123d
YARN-11313. [Federation] Add SQLServer Script and Supported DB Version in Federation.md. (#4927) 2022-10-05 13:25:20 -07:00
slfan1989
a708ff96f1
YARN-11324. Fix some PBImpl classes to avoid NPE. (#4961) 2022-10-04 14:30:20 -04:00
slfan1989
874a004347
YARN-11318. Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils. (#4939) 2022-10-04 09:18:01 -07:00
slfan1989
22bd5e3b53
YARN-11238. Optimizing FederationClientInterceptor Call with Parallelism. (#4904) 2022-10-04 09:17:00 -07:00
Riya Khandelwal
07581f1ab2
YARN-6169 message on empty configuration file improved (#4952) 2022-10-03 23:31:06 -04:00
Navink
4891bf5049
HDFS-13369. Fix for FSCK Report broken with RequestHedgingProxyProvider (#4917) 
Contributed-by: navinko <nakumr@cloudera.com>
 2022-09-30 23:28:12 +08:00
Mukund Thakur
e22f5e75ae
HADOOP-18463. Add an integration test to process data asynchronously during vectored read. (#4921) 
part of HADOOP-18103.

Contributed by: Mukund Thakur
 2022-09-28 23:16:47 +05:30
slfan1989
42d883937d
YARN-11310. [Federation] Refactoring Yarn Router's Federation Web Page. (#4924) 2022-09-27 13:31:52 -07:00
slfan1989
bfd6415827
YARN-11290. Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster. (#4846) 2022-09-27 13:28:52 -07:00
Mukund Thakur
735e35d648
HADOOP-18347. S3A Vectored IO to use bounded thread pool. (#4918) 
part of HADOOP-18103.

Also introducing a config fs.s3a.vectored.active.ranged.reads
to configure the maximum number of number of range reads a
single input stream can have active (downloading, or queued)
to the central FileSystem instance's pool of queued operations.
This stops a single stream overloading the shared thread pool.

Contributed by: Mukund Thakur
 2022-09-27 21:13:07 +05:30
Ashutosh Gupta
d9f435f6ac
HDFS-16766. XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source (#4886) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-27 15:44:25 +09:00
slfan1989
5d20988f9f
YARN-11308. Router Page display the db username and password in mask mode. (#4908) 2022-09-26 15:54:17 -07:00
slfan1989
0e65f4cc04
YARN-11316. Fix Yarn federation.md table format. (#4935) 2022-09-26 15:47:08 -07:00
slfan1989
aeba204fa2
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897) 2022-09-26 15:46:06 -07:00
Viraj Jasani
648071e197
HADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field (#4926) 
Follow-on to HADOOP-18455.

Contributed by Viraj Jasani
 2022-09-26 18:56:58 +01:00
Ashutosh Gupta
7923cac86b
HADOOP-18443. Upgrade snakeyaml to 1.32 (#4906) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:49:48 +09:00
Ashutosh Gupta
603e9bd745
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:30:58 +09:00
Ashutosh Gupta
fd0415c44a
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:26:06 +09:00
Xing Lin
747fb92107
HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash (#4869) 
* HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash

Signed-off-by: Xing Lin <xinglin@linkedin.com>
 2022-09-23 10:37:51 -07:00
Steve Loughran
0676495950
HADOOP-18456. NullPointerException in ObjectListingIterator. (#4909) 
This problem surfaced in impala integration tests
   IMPALA-11592. TestLocalCatalogRetries.test_fetch_metadata_retry fails in S3 build
after the change
  HADOOP-17461. Add thread-level IOStatistics Context
The actual GC race condition came with
 HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

The fix for this is, if our hypothesis is correct, in WeakReferenceMap.create()
where a strong reference to the new value is kept in a local variable
*and referred to later* so that the JVM will not GC it.

Along with the fix, extra assertions ensure that if the problem is not fixed,
applications will fail faster/more meaningfully. 

Contributed by Steve Loughran.
 2022-09-23 09:54:31 +01:00
Kidd5368
9a29075f91
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901) 2022-09-23 12:27:56 +09:00
slfan1989
e526f48fa4
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858) 2022-09-22 16:40:47 -07:00
PJ Fanning
e6d2c336cb
HADOOP-18341: upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 (#4578) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-22 09:45:20 +09:00