big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,601 Commits 2 Branches 2 Tags 264 MiB
1f61944e3b
Commit Graph

24601 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
df29aaaf80
YARN-10752. Shaded guava not found when compiling with profile hbase2.0. (#2957) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ic9943d18d4086b075ceed7924dcfa566653a16cb
 2021-04-27 00:41:42 +09:00
Stephen O'Donnell
f6efb58b07 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c29)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-04-26 11:23:56 +01:00
Wei-Chiu Chuang
1425512032 Revert "HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952)" 
This reverts commit d525e1e33d.
 2021-04-26 16:42:44 +08:00
Wei-Chiu Chuang
d525e1e33d HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 66dda86fec)
 2021-04-26 14:07:35 +08:00
Takanobu Asanuma
a5f038b3f6 HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 
(cherry picked from commit 01bad0e92a)
 2021-04-26 13:31:51 +09:00
Brahma Reddy Battula
99dfd3b2d0 HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2621d3f15b)
 2021-04-26 11:40:16 +08:00
Steve Loughran
8308aab658
HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

Change-Id: I2746aabcfeb0fbb138a80b02c4d5bbf2a8cf75da
 2021-04-25 18:43:25 +01:00
Steve Loughran
fb71e6c91e
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.

Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
 2021-04-24 18:32:39 +01:00
Gabor Bota
1b2bc77923
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593) 
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, because the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
 2021-04-24 18:28:21 +01:00
Mukund Thakur
33f9ceb3cc HADOOP-17136. ITestS3ADirectoryPerformance.testListOperations failing (#2153) 
A regression caused by HADOOP-17022: the reduction in LIST calls broken an assertion.

Contributed by Mukund Thakur

Change-Id: Ib9725165906931634567fd1f62a81e3a6ea5620c
 2021-04-24 18:28:14 +01:00
Mehakmeet Singh
389d3034c6
HADOOP-17471. ABFS to collect IOStatistics (#2731) (#2950) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-24 17:59:26 +01:00
Steve Loughran
77fddcfcb1
HADOOP-17535. ABFS: ITestAzureBlobFileSystemCheckAccess test failure if no oauth key. (#2920) 
Contributed by Steve Loughran.

Change-Id: I165f5ed3a8486404403827b5c0338cf7f80c2bb1
 2021-04-24 17:24:15 +01:00
Eric Badger
003deeeecf YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 22:14:57 +00:00
Jim Brennan
3b1cde2f42 YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 
(cherry picked from commit 1cbe35946b)
 2021-04-23 21:38:02 +00:00
Eric Badger
930f384e30 MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c)
 2021-04-23 18:42:50 +00:00
Ayush Saxena
b743d56eb4 HADOOP-17620. DistCp: Use Iterator for listing target directory as well. (#2861). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-04-23 22:49:28 +05:30
Akira Ajisaka
842714d99f
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0b04c9694d)
 2021-04-23 19:53:30 +09:00
Viraj Jasani
96a1dfa313
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 12:57:23 +09:00
zhuxiangyi
2752156e3e HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit dff95c5eca)
 2021-04-23 11:22:52 +09:00
zhangshuyan0
9f2db2c9fd
HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-21 18:32:56 +08:00
litao
29414871bd
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2928) 2021-04-21 10:41:13 +09:00
billierinaldi
8170a7bb60 HADOOP-16948. Support infinite lease dirs (#1925). Contributed by Billie Rinaldi. 
(cherry picked from commit c1fde4fe94)
 2021-04-20 14:36:54 -04:00
Eric Badger
1960924d07 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7)
 2021-04-20 17:46:05 +00:00
hemanthboyina
90f63cb12c HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 
(cherry picked from commit 16a20503ca)
 2021-04-20 16:21:57 +01:00
Steve Loughran
f30a0debae HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937) 
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
 2021-04-20 15:37:18 +01:00
Ayush Saxena
c93a2f6789 HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 
(cherry picked from commit c84e6beada)
 2021-04-20 12:13:39 +01:00
litao
dc830cf277 HDFS-15970. Print network topology on the web (#2896) 
(cherry picked from commit 85a3532849)
 2021-04-19 21:22:47 +09:00
Eric Badger
f7e382c62e YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 
(cherry picked from commit 31477a224f)
 2021-04-16 20:21:01 +00:00
Viraj Jasani
49f6326a9f HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-16 12:41:06 +05:30
sumangala-patki
8daa26d2e5
HADOOP-17576. ABFS: Disable throttling update for auth failures (#2761) (#2885) 
Contributed by Sumangala Patki

(cherry picked from commit 6f640abbaf)
 2021-04-16 10:47:11 +05:30
Hui Fei
4c0b833bab HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 
(cherry picked from commit 0d42104119)
 2021-04-16 14:07:15 +09:00
Masatake Iwasaki
a3405cb56a HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f85)
 2021-04-15 12:39:07 +00:00
He Xiaoqiao
a3cff8776a
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C. 2021-04-14 12:28:34 +08:00
Akira Ajisaka
72355c7b6e
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be)
 2021-04-13 17:10:00 +09:00
Wei-Chiu Chuang
bfba6f1f3c
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-12 23:55:32 -07:00
Akira Ajisaka
77315abe47
HADOOP-17608. Fix TestKMS failure (#2880) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a507)
 2021-04-12 12:54:59 +09:00
dependabot[bot]
d0b02a23f3 Bump y18n 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit 1a90fc4b50068865359809cd437e45595c397195)
 2021-04-11 01:31:27 +09:00
Eric Badger
1658a5140a YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 17:51:01 +00:00
Stephen O'Donnell
0e6cd352b5
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. (#2813) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2021-04-09 10:25:14 +01:00
lichaojacobs
068f114066
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x (#2775) 
(cherry picked from commit 663ca14a76)
 2021-04-09 12:00:38 +09:00
Viraj Jasani
4994b73eeb HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 
(cherry picked from commit 56bd968fb4)
 2021-04-09 11:13:39 +09:00
He Xiaoqiao
df99ac0399 HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df41997)
 2021-04-08 13:15:37 +01:00
Stephen O'Donnell
ef95f7a963 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7)
 2021-04-08 12:15:31 +01:00
touchida
dca2bf9dd5 HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 
(cherry picked from commit 95e6892675)
 2021-04-08 17:20:08 +08:00
Wei-Chiu Chuang
7f7535534d HADOOP-17625. Update to Jetty 9.4.39. (#2870) 
Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99)
 2021-04-08 14:28:03 +08:00
Brahma Reddy Battula
236a9a7713 HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree 
(cherry picked from commit ae88174c29)
 2021-04-07 23:50:02 +05:30
Gautham B A
b59bde61f6
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a)
 2021-04-07 19:58:03 +09:00
Wei-Chiu Chuang
1110b03752 HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282)
 2021-04-07 13:17:40 +08:00
Wei-Chiu Chuang
44365244e6 HADOOP-17624. Remove any rocksdb exclusion code. (#2867) 
Reviewed-by: Márton Elek <elek@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit f608cda729)
 2021-04-07 12:53:34 +08:00
Eric Badger
fb5809984e YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-06 23:34:35 +00:00