big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,949 Commits 2 Branches 2 Tags
Commit Graph

24949 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Masatake Iwasaki
76393e1359 HADOOP-17899. Avoid using implicit dependency on junit-jupiter-api. (#3399) 
(cherry picked from commit ce7a5bfbd3cb55afda265d105ff10ba2e2874a3f)
 2021-09-08 09:11:39 +00:00
Masatake Iwasaki
5926ccde77 HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 
(cherry picked from commit e183ec8998d0272884b73df7eb1a6da5adf1040a)
 2021-09-08 04:59:05 +00:00
Mukund Thakur
3b1c594355 HADOOP-17156. ABFS: Release the byte buffers held by input streams in close() (#3285) 
Contributed By: Mukund Thakur
 2021-09-07 15:29:22 +05:30
Yellow Flash
09e8e5c5cb
HADOOP-17870. Http Filesystem to qualify relative paths. (#3338) 
Contributed by Yellowflash

Change-Id: I217da06a1a2e5c0ca2b324f8e21baa0846f64858
 2021-09-07 10:54:35 +01:00
Chris Nauroth
cc90b4f987 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 18:48:07 +00:00
Viraj Jasani
7a4eaeb8bf
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99a157fa4ad174c4ff979414af2edb67a98eb9fe)
 2021-09-03 10:27:06 +09:00
Ahmed Hussein
fc5d67dfb4 HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
 2021-09-02 16:17:32 -05:00
lzx404243
d2c02f5afc
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
(cherry picked from commit 6187f76f11e885023c99382f0fd18f8fea8bcbb6)
 2021-09-01 17:15:31 +09:00
Szilard Nemeth
0a726250ea
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f76e470527ad94b0894dacb28db01465)
 2021-09-01 10:44:15 +09:00
Uma Maheswara Rao G
580b6c400b
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
(cherry picked from commit 164608b5465ebc6a2d296376aafbf4b272d9cff0)
 2021-08-31 12:27:43 +08:00
Dongjoon Hyun
8606b2cddd
HADOOP-17869. fs.s3a.connection.maximum should be bigger than fs.s3a.threads.max (#3337). 
The value of `fs.s3a.connection.maximum` has been increased to 96

Contributed by Dongjoon Hyun

Change-Id: I9020a2bfd2a67fa7a2ec0598ed9d63e78ee99c73
 2021-08-30 18:31:57 +01:00
lzx404243
4a93ca78f9
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() (#2968) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7b5be74228cb4a68d11f52ac061829b70a4f3144)
 2021-08-30 08:41:46 +09:00
jianghuazhu
7c663043b2
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364e9258247029c22a222a665771ab4c0)
 2021-08-27 12:06:26 +08:00
jianghuazhu
2b2f8f575b
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit ad54f5195c8c01f333703c55cd70703109d75f29)
 2021-08-25 17:35:50 +08:00
Viraj Jasani
0967483b7c HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 1b9927afe1dba136ea460ec1417514c0d7815504)
 2021-08-25 17:42:56 +09:00
Masatake Iwasaki
3645a13586 HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 
(cherry picked from commit 0d59500e8cf89e38cd0f0e45553e4557e5358a8b)
 2021-08-25 01:54:36 +00:00
Viraj Jasani
fc6b1cafd4 HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 
(cherry picked from commit fc566ad9b0e1336381c5f6b321cdcff81768f47a)
 2021-08-24 22:44:11 +09:00
Szilard Nemeth
224b42108d YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. Contributed by Tamas Domok 2021-08-24 14:16:15 +02:00
litao
5a1ed37893
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e2bf4c87f12b53e508edf8fee05856a)
 2021-08-19 12:31:52 +08:00
Siyao Meng
226f94b4fc HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3aaac8a1f682ecda4d41221e8a836bc5401e3b1d)
 2021-08-16 00:32:05 +09:00
Renukaprasad C
5b566c3914 HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b90389ae9840ccfdcab411e286547895a647feed)

 Conflicts:
	LICENSE-binary
 2021-08-14 20:00:38 +09:00
jianghuazhu
0a5f76b814
HDFS-16151. Improve the parameter comments related to ProtobufRpcEngine2#Server(). (#3256) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c7b951e031d60310acc7b4b1aa5052624fcfb95)
 2021-08-08 14:57:08 +09:00
Akira Ajisaka
025ecf42be
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 
(cherry picked from commit 3565c9477dffd104a3aa3603845f7a1305815ac3)
 2021-08-08 11:25:26 +09:00
Akira Ajisaka
00c382d118
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:27:52 +09:00
Bryan Beaudreault
2fda130260 HADOOP-17837: Add unresolved endpoint value to UnknownHostException (ADDENDUM) (#3276) 
(cherry picked from commit b0b867e977ab853d1dfc434195c486cf0ca32dab)
 2021-08-06 21:57:46 +05:30
wangzhaohui
7068e18842
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c4d1368e1195132bb32a815cb12934c)
 2021-08-06 18:25:00 +08:00
Bryan Beaudreault
7659b62682
HADOOP-17837: Add unresolved endpoint value to UnknownHostException (#3272) 
(cherry picked from commit 5e54d92e6ec866dc49a750110863a3fa8b2bcf7c)
 2021-08-06 17:32:01 +08:00
wangzhaohui
e9ba4f4591 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b733177cf9b8b10088527e34099d73f)
 2021-08-06 09:28:39 +01:00
Viraj Jasani
b3077543cf HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec11c3e1c2ed5ee9db801d40a2f7a2ed6)
 2021-08-06 09:30:43 +08:00
Siyao Meng
72508e6430 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00b5589fe610ff994d81feba86039297)
 2021-08-03 20:24:18 +01:00
Steve Loughran
c1ad91e72d
HADOOP-17822. fs.s3a.acl.default not working after S3A Audit feature (#3249) 
Fixes the regression caused by HADOOP-17511 by moving where the
option  fs.s3a.acl.default is read -doing it before the RequestFactory
is created.

Adds

* A unit test in TestRequestFactory to verify the ACLs are set
  on all file write operations.
* A new ITestS3ACannedACLs test which verifies that ACLs really
  do get all the way through.
* S3A Assumed Role delegation tokens to include the IAM permission
  s3:PutObjectAcl in the generated role.

Contributed by Steve Loughran

Change-Id: I3abac6a1b9e150b6b6df0af7c2c70093f8f518cb
 2021-08-02 15:33:34 +01:00
Steve Loughran
26514b6534 HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 
This patch cuts down the size of directory trees used for
distcp contract tests against object stores, so making
them much faster against distant/slow stores.

On abfs, the test only runs with -Dscale (as was the case for s3a already),
and has the larger scale test timeout.

After every test case, the FileSystem IOStatistics are logged,
to provide information about what IO is taking place and
what it's performance is.

There are some test cases which upload files of 1+ MiB; you can
increase the size of the upload in the option
"scale.test.distcp.file.size.kb" 
Set it to zero and the large file tests are skipped.

Contributed by Steve Loughran.
 2021-08-02 12:58:37 +01:00
Bobby Wang
904cdd0b00
HADOOP-17812. NPE in S3AInputStream read() after failure to reconnect to store (#3222) 
This improves error handling after multiple failures reading data
-when the read fails and attempts to reconnect() also fail.

Contributed by Bobby Wang.

Change-Id: If17dee395ad6b9b7c738021bad20d0a13eb4011e
 2021-08-02 12:58:25 +01:00
Petre Bogdan Stolojan
f2cec5cb88
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan

Change-Id: I25d502102775c3626c4264e5a14c649879730050
 2021-08-02 11:58:36 +01:00
Brian Loss
37e0828e76
HADOOP-17811: ABFS ExponentialRetryPolicy doesn't pick up configuration values (#3221) 
Contributed by Brian Loss.

Change-Id: I5f24196d1d02de91336c3679abaf8d55cfaed746
 2021-08-02 11:37:33 +01:00
jenny
b8a8821735
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c8641384fa4b45d8a27c4b7a152c1ea35b)
 2021-08-02 15:39:38 +09:00
Szilard Nemeth
0a0628b1cb YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:53:30 +02:00
Szilard Nemeth
a272adc5fa YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-07-29 19:21:58 +02:00
He Xiaoqiao
797c595ba5
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4863499207733a9f4c8c3cd2ec9c9d2)
 2021-07-29 19:04:08 +08:00
jianghuazhu
43f15578fc
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 6f730fd25c9845fea09b98c64b5d59bee5576db2)
 2021-07-29 15:44:02 +08:00
hchaverr
6cc1426b63 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 15:48:51 -07:00
9uapaw
f001595b47 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
 2021-07-28 17:12:40 +02:00
S O'Donnell
d661afc06f HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 
(cherry picked from commit 4eae284827c040cd9fd0eab2e7fd8aaae721326e)
 2021-07-28 15:02:16 +01:00
Szilard Nemeth
72801be13a YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-07-28 14:55:19 +02:00
bshashikant
18bd66e5b0 HDFS-16145. CopyListing fails with FNF exception with snapshot diff. (#3234) 
(cherry picked from commit dac10fcc202ed6d1fe4bd852f57a6bbcbadd90fe)
 2021-07-28 09:38:06 +01:00
Akira Ajisaka
9e72b697a2
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit d710ec8d85692e4c2f915db1474b15dba84af666)
 2021-07-26 16:47:26 +09:00
dependabot[bot]
e533efc58e [UI2] Bump http-proxy to 1.18.1 (#2891) 
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b7431c3ed5e1fff947e84f3a4847b6d2864658d8)
 2021-07-26 02:17:36 +09:00
Akira Ajisaka
3f573b6751 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60ceed733bfacb791fc5ed06116720dd0.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488c8da0227b95548c1fe40823b16a44d)
 2021-07-26 01:54:32 +09:00
Petre Bogdan Stolojan
e89d30b6b7
HADOOP-17458. S3A to treat "SdkClientException: Data read has a different length than the expected" as EOFException (#3040) 
Some network exceptions can raise SdkClientException with message
`Data read has a different length than the expected`.

These should be recoverable.

Contributed by Bogdan Stolojan

Change-Id: Ia22fd77d90971e9e02b4f947398a4749eebe5909
 2021-07-23 14:46:59 +01:00
Viraj Jasani
5ea1337167 HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db1b13a71d5a5c6ee0cd689fb8b27895)
 2021-07-23 12:42:52 +08:00