big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,042 Commits 2 Branches 2 Tags 264 MiB
cb11fd8793
Commit Graph

24042 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 14:52:05 -08:00
Akira Ajisaka
dd1634ec3b HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04c)
 2020-11-03 11:22:48 -08:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518)
 2020-10-29 19:52:43 +00:00
Sunil G
91a3d298b9 HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d52)
 2020-10-29 07:20:46 +05:30
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743)
 2020-10-28 10:33:41 -07:00
Brahma Reddy Battula
fe8b22ca8f HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1)
 2020-10-28 09:28:09 -07:00
Akira Ajisaka
8e0922edb2 HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 
(cherry picked from commit b3ba74d72d)
 2020-10-28 19:14:19 +09:00
Ahmed Hussein
d4129c6794
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4)
 2020-10-27 20:49:15 +09:00
Ahmed Hussein
1429c65533
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d332)
 2020-10-27 16:58:04 +09:00
Ayush Saxena
af5f90623c HADOOP-17328. LazyPersist Overwrite fails in direct write mode. (#2413) 
(cherry picked from commit 872440610f)
 2020-10-27 01:40:25 +09:00
Hui Fei
b8cc09d7c7 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:25:03 +08:00
Anoop Sam John
8312f230eb HADOOP-17308. WASB PageBlobOutputStream.flush succeeds even when flush to storage fails (#2392) 
Contributed by Anoop Sam John.
 2020-10-26 13:31:53 +00:00
Wei-Chiu Chuang
c8eb86ee3d HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
 2020-10-23 12:06:23 -07:00
Inigo Goiri
c55d639e8f HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a2)
 2020-10-23 09:48:23 -07:00
Vinayakumar B
e21b81276e
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:34:14 +05:30
Kihwal Lee
02709cb054 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 19:41:38 -05:00
Akira Ajisaka
c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 
(cherry picked from commit 7bc305db5d)
 2020-10-23 04:41:37 +09:00
Kihwal Lee
cf932a7e2d HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-22 09:55:28 -05:00
Takanobu Asanuma
0bb1f0df27 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:12:26 +09:00
tkuramoto33
0289cdc43c
HADOOP-17298. Backslash in username causes build failure in the environment started by start-build-env.sh. (#2367) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a2e7a073f0)
 2020-10-20 10:11:18 +09:00
Akira Ajisaka
37c2e34b02
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb42)
 2020-10-19 10:49:21 +09:00
Ayush Saxena
54c40cbf49
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383) 
Contributed by Gabor Bota.
 2020-10-17 01:34:01 +05:30
Kihwal Lee
433434d509 HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:16:47 -05:00
Eric Badger
8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780)
 2020-10-15 18:09:00 +00:00
Sneha Vijayarajan
d5b4d04b0d
HADOOP-17301. ABFS: read-ahead error reporting breaks buffer management (#2369) 
Fixes read-ahead buffer management issues introduced by HADOOP-16852,
 "ABFS: Send error back to client for Read Ahead request failure".

Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
da5db6a5a6
HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. 
Contributed by Sneha Vijayarajan

Testing:

namespace.enabled=false
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 246
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 33
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
d166420302
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
bilaharith
f208da286c
HADOOP-17166. ABFS: configure output stream thread pool (#2179) 
Adds the options to control the size of the per-output-stream threadpool
when writing data through the abfs connector

* fs.azure.write.max.concurrent.requests
* fs.azure.write.max.requests.to.queue

Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
cc7350302f
HADOOP-16915. ABFS: Ignoring the test ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
4072323de4
Upgrade store REST API version to 2019-12-12 
- Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
bilaharith
e481d0108a
HADOOP-17149. ABFS: Fixing the testcase ITestGetNameSpaceEnabled 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
f73c90f0b0
HADOOP-17163. ABFS: Adding debug log for rename failures 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
fbf151ef6f
HADOOP-17137. ABFS: Makes the test cases in ITestAbfsNetworkStatistics agnostic 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
Kihwal Lee
41a3c9bc95 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d)
 2020-10-14 17:28:02 -05:00
Mukund Thakur
351b1498d3
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:32:35 +01:00
Ayush Saxena
2a043b987c
HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 
(cherry picked from commit 7a6265ac42)
 2020-10-13 18:29:54 +09:00
Pranav Bheda
054dba68c2
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 
* update org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.13
* update org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13

(cherry picked from commit be3edd0532)
 2020-10-13 17:56:38 +09:00
Akira Ajisaka
2e73871cab
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 
(cherry picked from commit 69ef9b1ee8)
 2020-10-13 17:43:29 +09:00
Konstantin V Shvachko
b6423d2780 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:38:42 -07:00
Akira Ajisaka
800b1ed1c2 Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-12 11:59:17 -07:00
Swaroopa Kadam
05a73ded93
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:09:08 -07:00
Dongjoon Hyun
5032f8abba
HADOOP-17258. Magic S3Guard Committer to overwrite existing pendingSet file on task commit (#2371) 
Contributed by Dongjoon Hyun and Steve Loughran

Change-Id: Ibaf8082e60eff5298ff4e6513edc386c5bae0274
 2020-10-12 13:42:08 +01:00
Steve Loughran
7cf5bdeec0 Revert "HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375)" 
This reverts commit 263b7d5dfc.
 2020-10-12 10:45:18 +01:00
Akira Ajisaka
263b7d5dfc HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 10:43:26 +01:00
Doroszlai, Attila
13e0c5f6e0 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-08 23:44:56 -07:00
Jim Brennan
76e223a320 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32ba)
 2020-10-08 18:59:25 +00:00
Steve Loughran
963793dd48
HADOOP-17293. S3A to always probe S3 in S3A getFileStatus on non-auth paths 
This reverts changes in HADOOP-13230 to use S3Guard TTL in choosing when
to issue a HEAD request; fixing tests to compensate.

New org.apache.hadoop.fs.s3a.performance.OperationCost cost,
S3GUARD_NONAUTH_FILE_STATUS_PROBE for use in cost tests.

Contributed by Steve Loughran.

Change-Id: I418d55d2d2562a48b2a14ec7dee369db49b4e29e
 2020-10-08 15:38:32 +01:00
Jinglun
44ff4c1058
HADOOP-17021. Add concat fs command (#1993) 
Contributed by Jinglun

Change-Id: Ia10ad2205ed0f3594c391ee78f7df4c3c31c796d
 2020-10-08 10:36:40 +01:00