big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
26,182 Commits 2 Branches 2 Tags 264 MiB
aeba204fa2
Commit Graph

26182 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
slfan1989
aeba204fa2
YARN-11306. [Federation] Refactor NM#FederationInterceptor#recover Code. (#4897) 2022-09-26 15:46:06 -07:00
Viraj Jasani
648071e197
HADOOP-18466. Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field (#4926) 
Follow-on to HADOOP-18455.

Contributed by Viraj Jasani
 2022-09-26 18:56:58 +01:00
Ashutosh Gupta
7923cac86b
HADOOP-18443. Upgrade snakeyaml to 1.32 (#4906) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:49:48 +09:00
Ashutosh Gupta
603e9bd745
YARN-11271. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-common (#4774) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:30:58 +09:00
Ashutosh Gupta
fd0415c44a
YARN-11270. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice-hbase-client (#4773) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-25 23:26:06 +09:00
Xing Lin
747fb92107
HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash (#4869) 
* HADOOP-18444 Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash

Signed-off-by: Xing Lin <xinglin@linkedin.com>
 2022-09-23 10:37:51 -07:00
Steve Loughran
0676495950
HADOOP-18456. NullPointerException in ObjectListingIterator. (#4909) 
This problem surfaced in impala integration tests
   IMPALA-11592. TestLocalCatalogRetries.test_fetch_metadata_retry fails in S3 build
after the change
  HADOOP-17461. Add thread-level IOStatistics Context
The actual GC race condition came with
 HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

The fix for this is, if our hypothesis is correct, in WeakReferenceMap.create()
where a strong reference to the new value is kept in a local variable
*and referred to later* so that the JVM will not GC it.

Along with the fix, extra assertions ensure that if the problem is not fixed,
applications will fail faster/more meaningfully. 

Contributed by Steve Loughran.
 2022-09-23 09:54:31 +01:00
Kidd5368
9a29075f91
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task (#4901) 2022-09-23 12:27:56 +09:00
slfan1989
e526f48fa4
YARN-11296. [Federation] Fix SQLFederationStateStore#Sql script bug. (#4858) 2022-09-22 16:40:47 -07:00
PJ Fanning
e6d2c336cb
HADOOP-18341: upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 (#4578) 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-22 09:45:20 +09:00
Viraj Jasani
084b68e380
HADOOP-18455. S3A prefetching executor should be closed (#4879) 
follow-on patch to HADOOP-18186. 

Contributed by: Viraj Jasani
 2022-09-22 00:22:41 +05:30
Samrat
740e1ef357
HDFS-16706. ViewFS doc points to wrong mount table name (#4803) 2022-09-21 16:55:20 +05:30
Ashutosh Gupta
917aef75fc
YARN-11255. Support loading alternative docker client config from system environment (#4884) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-09-21 16:54:02 +05:30
slfan1989
4c5a7cc6fc
YARN-11307. Fix Yarn Router Broken Link. (#4905) 2022-09-20 23:10:09 -04:00
slfan1989
4d9bb81b16
HADOOP-18451. Update hsqldb.version from 2.3.4 to 2.5.2. (#4880) 2022-09-20 11:10:51 -07:00
GuoPhilipse
40ab9c8ba7
HDFS-16341. Fix BlockPlacementPolicy details in hdfs defaults. (#3691). Contributed by guophilipse. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:16:24 +05:30
Ashutosh Gupta
2950c5405b
HADOOP-16674. Fix when TestDNS.testRDNS can fail with ServiceUnavailableException (#4802). Contributed by Ashutosh Gupta. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:09:59 +05:30
slfan1989
fd687bb4c4
YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241. (#4893). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-20 23:06:24 +05:30
Ashutosh Gupta
a134628d1b
YARN-11303. Upgrade jquery ui to 1.13.2 to mitigate CVE-2022-31160 (#4895) 2022-09-20 09:50:03 -07:00
Viraj Jasani
5b1657278c
HADOOP-18377. hadoop-aws build to add a -prefetch profile to run all tests with prefetching (#4914) 
Contributed by Viraj Jasani
 2022-09-20 10:26:13 +01:00
ZanderXu
e68006cd70
HDFS-16772. refreshHostsReader should use the latest configuration (#4890) 2022-09-19 13:27:07 -07:00
slfan1989
f52b900a5f
YARN-11283. Fix Typo of NodeManager amrmproxy. (#4899) 2022-09-19 13:16:25 -07:00
slfan1989
342c4856b8
YARN-11293. [Federation] StoreNewMasterKey/removeStoredMasterKey With MemoryStateStore. (#4852) 2022-09-19 13:14:55 -07:00
GuoPhilipse
620dd37712
HADOOP-18118. [Follow on] Fix test failure in TestHttpServer (#4900) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2022-09-19 09:10:00 -07:00
Ashutosh Gupta
30c36ef25a
HADOOP-18400. Fix file split duplicating records from a succeeding split when reading BZip2 text files (#4732) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-19 13:45:05 +09:00
ZanderXu
a73c4804d8
HADOOP-18446. [SBN read] Add a re-queue metric to RpcMetrics to quantify the number of re-queued RPCs (#4871) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-16 10:09:01 -07:00
Ashutosh Gupta
0f03299eba
HADOOP-16769. LocalDirAllocator to provide diagnostics when file creation fails (#4842) 
The patch provides detailed diagnostics of file creation failure in LocalDirAllocator.

Contributed by: Ashutosh Gupta
 2022-09-16 13:17:26 +05:30
ZanderXu
43c1ebae16
HDFS-16771. Follow-up for HDFS-16659: JN should tersely print logs about NewerTxnIdException (#4882) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
 2022-09-15 12:44:36 -07:00
Ashutosh Gupta
59d3c20118
MAPREDUCE-7407. Avoid stopContainer() on dead node (#4779) 2022-09-15 10:30:36 -07:00
Simbarashe Dzinamarira
6422eaf301
HDFS-16767: RBF: Support observer node in Router-Based Federation. 
Fixes #4127

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-14 16:00:14 -07:00
GuoPhilipse
ce54b7e55d
HADOOP-18118. Fix KMS Accept Queue Size default value to 500 (#3972) 2022-09-15 00:53:25 +08:00
Colm O hEigeartaigh
272844ee57
HADOOP-15072 - Update Apache Kerby to 2.0.2 (#4473) 2022-09-15 00:43:25 +08:00
slfan1989
86b84ed74e
HADOOP-18452. Fix TestKMS#testKMSHAZooKeeperDelegationToken Failed By Hadoop-18427. (#4885) 2022-09-14 09:13:58 -07:00
Renukaprasad C
8ce71c8882
[Documentation] RBF: Duplicate statement to be removed for better readabilty (#4881) 2022-09-13 09:24:05 -07:00
slfan1989
88545a875d
YARN-11301. Fix NoClassDefFoundError: org/junit/platform/launcher/core/LauncherFactory after YARN-11269. (#4878) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:08:16 +09:00
Ashutosh Gupta
55d8a91b2c
YARN-11261. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy (#4777) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-13 17:05:24 +09:00
slfan1989
3ce353395b
YARN-7614. [RESERVATION] Support ListReservation APIs in Federation Router. (#4843) 2022-09-12 12:33:21 -07:00
Ashutosh Gupta
65a027b112
YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled (#4703) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-12 23:32:20 +09:00
slfan1989
cde1f3af21
HADOOP-18302. Remove WhiteBox in hadoop-common module. (#4457) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-12 23:28:16 +09:00
caozhiqiang
1923096adb
Allow block reconstruction pending timeout to be refreshable (#4567) 
Reviewed-by: Hiroyuki Adachi <hadachi@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-09-12 11:45:01 +09:00
Ashutosh Gupta
21bae31d58
YARN-11265. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-sharedcachemanager (#4772) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-09-11 03:31:28 +09:00
slfan1989
cdcb448b78
YARN-11286. Make AsyncDispatcher#printEventDetailsExecutor thread pool parameter configurable. (#4824). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 23:00:50 +05:30
slfan1989
b2760520c3
YARN-11274. Impove Nodemanager#NodeStatusUpdaterImpl Log. (#4783). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-10 22:57:06 +05:30
Simbarashe Dzinamarira
e77d54d1ee
HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients. 
Fixes #4311

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-09-09 15:43:28 -07:00
slfan1989
e76ffbf102
YARN-11297. [Federation] Improve Yarn Router Reservation Submission Code. (#4863) 2022-09-09 10:39:00 -07:00
Mukund Thakur
8732625f50
HADOOP-18439. Fix VectoredIO for LocalFileSystem when checksum is enabled. (#4862) 
part of HADOOP-18103.

While merging the ranges in CheckSumFs, they are rounded up based on the
value of checksum bytes size which leads to some ranges crossing the EOF
thus they need to be fixed else it will cause EOFException during actual reads.

Contributed By: Mukund Thakur
 2022-09-09 21:46:08 +05:30
9uapaw
5b85af87f0 YARN-11278. Fixed Ambiguous error message in mutation API. Contributed by Ashutosh Gupta. 2022-09-09 14:38:41 +02:00
Viraj Jasani
56387cce57
HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work (#4796) 
Contributed by Viraj Jasani
 2022-09-09 11:32:20 +01:00
ZanderXu
4a01fadb94
HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-09-09 12:47:36 +05:30
slfan1989
29c4d8d8f7
YARN-11298. Improve Yarn Router Junit Test Close MockRM. (#4870) 2022-09-08 11:42:36 -07:00