big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,211 Commits 2 Branches 2 Tags 264 MiB
9512c774c0
Commit Graph

25211 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files (#3318) 
* YARN 10884 : Parse log files which has empty owner

* Removing the whitespace

* Added a test case for null user

* Fixed indentations

* Fixed the indentation for test cases

Co-authored-by: Swathi C <cswathi@microsoft.com>
 2021-09-07 21:37:51 +05:30
Steve Loughran
6e3aeb1544
HADOOP-17894. CredentialProviderFactory.getProviders() recursion loading JCEKS file from S3A (#3393) 
* CredentialProviderFactory to detect and report on recursion.
* S3AFS to remove incompatible providers.
* Integration Test for this.

Contributed by Steve Loughran.
 2021-09-07 15:29:37 +01:00
Mukund Thakur
9b8f81a179
HADOOP-17156. ABFS: Release the byte buffers held by input streams in close() (#3285) 
Contributed By: Mukund Thakur
 2021-09-07 15:13:36 +05:30
jianghuazhu
4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368) 2021-09-06 18:06:12 +08:00
Renukaprasad C
dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244). Contributed by Renukaprasad C. 2021-09-05 17:07:59 +08:00
litao
364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354). Contributed by tomscut. 2021-09-05 14:19:01 +08:00
Viraj Jasani
cc6b545365
HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-04 16:20:08 +05:30
Chris Nauroth
1d808f59d7 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 18:43:48 +00:00
Weison Wei
a610f6d9c3
HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367). Contributed by Weison Wei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-03 12:05:53 +05:30
Viraj Jasani
99a157fa4a
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-03 10:25:33 +09:00
Ahmed Hussein
051207375b
HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 2021-09-02 16:11:07 -05:00
langlaile1221
24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894) 2021-09-01 19:37:34 +08:00
lzx404243
6187f76f11
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
 2021-09-01 13:47:09 +09:00
Yellow Flash
4ea60b5733
HADOOP-17870. Http Filesystem to qualify relative paths. (#3338) 
Contributed by Yellowflash
 2021-08-31 13:55:52 +01:00
Uma Maheswara Rao G
164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2021-08-31 12:25:03 +08:00
Dongjoon Hyun
265a48e245
HADOOP-17869. fs.s3a.connection.maximum should be bigger than fs.s3a.threads.max (#3337). 
The value of `fs.s3a.connection.maximum` has been increased to 96

Contributed by Dongjoon Hyun
 2021-08-30 18:30:43 +01:00
Akira Ajisaka
50dda774f1
HADOOP-17544. Mark KeyProvider as Stable. (#2776) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-08-30 09:55:53 +09:00
lzx404243
7b5be74228
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() (#2968) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-30 08:39:33 +09:00
Liang-Chi Hsieh
73a0c31370
HADOOP-17877. BuiltInGzipCompressor header and trailer should not be static variables (#3350) 2021-08-29 08:21:55 -07:00
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-27 23:09:54 +02:00
jianghuazhu
4c94831364
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-27 11:41:44 +08:00
Viraj Jasani
aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
jianghuazhu
ad54f5195c
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-08-25 17:34:45 +08:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Gautham B A
c8e5864838
HADOOP-17854. Run junit in Jenkins only if surefire reports exist (#3319) 2021-08-24 10:26:26 -07:00
Viraj Jasani
fc566ad9b0
HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 2021-08-24 22:41:59 +09:00
Szilard Nemeth
e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
Neil
9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315) 2021-08-24 11:15:47 +08:00
Gautham B A
b6d1971820
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311) 2021-08-20 10:04:55 -07:00
litao
07627ef19e
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-08-19 12:30:38 +08:00
litao
0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291) 2021-08-19 09:51:00 +08:00
sumangala-patki
dcddc6a59f
HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters (#2975) 2021-08-18 19:14:10 +05:30
Steve Loughran
ee07b90286
HADOOP-17836. Improve logging on ABFS error reporting (#3281) 
Contributed by Steve Loughran.
 2021-08-18 11:39:17 +01:00
Gautham B A
409878ef80
HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303) 2021-08-17 09:49:52 -07:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287) 
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
 2021-08-17 14:18:55 +05:30
jianghuazhu
53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Liang-Chi Hsieh
6014a089fd
HADOOP-17825. Add BuiltInGzipCompressor (#3250) 
Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is not loaded. So, without Hadoop native codec installed, saving SequenceFile using GzipCodec will throw exception like "SequenceFile doesn't work with GzipCodec without native-hadoop code!"

Same as other codecs which we migrated to using prepared packages (lz4, snappy), it will be better if we support GzipCodec generally without Hadoop native codec installed. Similar to BuiltInGzipDecompressor, we can use Java Deflater to support BuiltInGzipCompressor.
 2021-08-16 10:08:03 -07:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani
919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Siyao Meng
3aaac8a1f6
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 21:19:10 +09:00
Akira Ajisaka
3ec53a33ba
HADOOP-17799. Improve the GitHub pull request template (#3277) 
Reviewed-by: Yi Sheng <yisheng@apache.org>
 2021-08-14 21:16:15 +09:00
Renukaprasad C
b90389ae98
HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-14 19:55:32 +09:00
Gautham B A
77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Viraj Jasani
6a7883431f
HADOOP-17841. Remove ListenerHandle from Hadoop registry (#3278) 2021-08-09 16:57:53 +08:00
jianghuazhu
0c7b951e03
HDFS-16151. Improve the parameter comments related to ProtobufRpcEngine2#Server(). (#3256) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-08 14:55:55 +09:00
Akira Ajisaka
3565c9477d
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 2021-08-08 11:23:28 +09:00
Akira Ajisaka
4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
Viraj Jasani
23e2a0b202
HADOOP-17835. Use CuratorCache implementation instead of PathChildrenCache / TreeCache (#3266) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-07 11:20:35 +09:00
Gautham B A
4fd97e01e5
HADOOP-17787. Refactor fetching of credentials in Jenkins (#3167) 2021-08-06 11:56:27 -07:00