big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,043 Commits 2 Branches 2 Tags
Commit Graph

24043 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
cfa0986d00 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit dd1634ec3b676d6c3da18af7da46734584ddd9f6.
 2020-11-04 16:18:23 -08:00
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4aa0d6cc9f9be530802e54610776a4d)
 2020-11-03 14:52:05 -08:00
Akira Ajisaka
dd1634ec3b HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04cb8e5dab8410ef64daa0fe799e389fd)
 2020-11-03 11:22:48 -08:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518bfdf7ad257cc1cf3c73f4208c49fc0)
 2020-10-29 19:52:43 +00:00
Sunil G
91a3d298b9 HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d527b0089124a8ca69b36fb20ed118c55)
 2020-10-29 07:20:46 +05:30
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
 2020-10-28 10:33:41 -07:00
Brahma Reddy Battula
fe8b22ca8f HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1e48db05ab827d0cf09963a430615ed)
 2020-10-28 09:28:09 -07:00
Akira Ajisaka
8e0922edb2 HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 
(cherry picked from commit b3ba74d72dfe938ea7ed9e6b0870db6c87aa0a91)
 2020-10-28 19:14:19 +09:00
Ahmed Hussein
d4129c6794
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac43a480a010d40c182c1d4cc5346db8f)
 2020-10-27 20:49:15 +09:00
Ahmed Hussein
1429c65533
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d3325e0b454475fa01cc50f68007b0e15c)
 2020-10-27 16:58:04 +09:00
Ayush Saxena
af5f90623c HADOOP-17328. LazyPersist Overwrite fails in direct write mode. (#2413) 
(cherry picked from commit 872440610f066d4b12c9f93c05477848a260b21f)
 2020-10-27 01:40:25 +09:00
Hui Fei
b8cc09d7c7 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827c80ea37ed5207a65831d0b699c8381)
 2020-10-26 22:25:03 +08:00
Anoop Sam John
8312f230eb HADOOP-17308. WASB PageBlobOutputStream.flush succeeds even when flush to storage fails (#2392) 
Contributed by Anoop Sam John.
 2020-10-26 13:31:53 +00:00
Wei-Chiu Chuang
c8eb86ee3d HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002717a9684d00f0e9dc263ab7eb49246)
 2020-10-23 12:06:23 -07:00
Inigo Goiri
c55d639e8f HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a26988fcf5880311067e62a15e05fc1e)
 2020-10-23 09:48:23 -07:00
Vinayakumar B
e21b81276e
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:34:14 +05:30
Kihwal Lee
02709cb054 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc286db00b385f3280627d2b2063b4e59)
 2020-10-22 19:41:38 -05:00
Akira Ajisaka
c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 
(cherry picked from commit 7bc305db5d5a50127dd7780534d6d7d0b7c683d1)
 2020-10-23 04:41:37 +09:00
Kihwal Lee
cf932a7e2d HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-22 09:55:28 -05:00
Takanobu Asanuma
0bb1f0df27 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:12:26 +09:00
tkuramoto33
0289cdc43c
HADOOP-17298. Backslash in username causes build failure in the environment started by start-build-env.sh. (#2367) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a2e7a073f03f682d6716ed59c60533be09841784)
 2020-10-20 10:11:18 +09:00
Akira Ajisaka
37c2e34b02
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb4207af2f06f8ef57e83ff9759871d0cb)
 2020-10-19 10:49:21 +09:00
Ayush Saxena
54c40cbf49
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383) 
Contributed by Gabor Bota.
 2020-10-17 01:34:01 +05:30
Kihwal Lee
433434d509 HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:16:47 -05:00
Eric Badger
8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780c1505ab7f302c3116a0c5cdd122315)
 2020-10-15 18:09:00 +00:00
Sneha Vijayarajan
d5b4d04b0d
HADOOP-17301. ABFS: read-ahead error reporting breaks buffer management (#2369) 
Fixes read-ahead buffer management issues introduced by HADOOP-16852,
 "ABFS: Send error back to client for Read Ahead request failure".

Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
da5db6a5a6
HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. 
Contributed by Sneha Vijayarajan

Testing:

namespace.enabled=false
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 246
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 33
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
d166420302
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
bilaharith
f208da286c
HADOOP-17166. ABFS: configure output stream thread pool (#2179) 
Adds the options to control the size of the per-output-stream threadpool
when writing data through the abfs connector

* fs.azure.write.max.concurrent.requests
* fs.azure.write.max.requests.to.queue

Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
cc7350302f
HADOOP-16915. ABFS: Ignoring the test ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
4072323de4
Upgrade store REST API version to 2019-12-12 
- Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
bilaharith
e481d0108a
HADOOP-17149. ABFS: Fixing the testcase ITestGetNameSpaceEnabled 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
f73c90f0b0
HADOOP-17163. ABFS: Adding debug log for rename failures 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
fbf151ef6f
HADOOP-17137. ABFS: Makes the test cases in ITestAbfsNetworkStatistics agnostic 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
Kihwal Lee
41a3c9bc95 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d4e9a9804c777c8f845ad41e1280177)
 2020-10-14 17:28:02 -05:00
Mukund Thakur
351b1498d3
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:32:35 +01:00
Ayush Saxena
2a043b987c
HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 
(cherry picked from commit 7a6265ac425e7ff1426c015e81f8ac882ff7bcf6)
 2020-10-13 18:29:54 +09:00
Pranav Bheda
054dba68c2
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 
* update org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.13
* update org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13

(cherry picked from commit be3edd05321a0cbe5e6d60536b4ab6ed34f2cef6)
 2020-10-13 17:56:38 +09:00
Akira Ajisaka
2e73871cab
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 
(cherry picked from commit 69ef9b1ee8825fbf6f06c2e3b84ca93e22398953)
 2020-10-13 17:43:29 +09:00
Konstantin V Shvachko
b6423d2780 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:38:42 -07:00
Akira Ajisaka
800b1ed1c2 Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-12 11:59:17 -07:00
Swaroopa Kadam
05a73ded93
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:09:08 -07:00
Dongjoon Hyun
5032f8abba
HADOOP-17258. Magic S3Guard Committer to overwrite existing pendingSet file on task commit (#2371) 
Contributed by Dongjoon Hyun and Steve Loughran

Change-Id: Ibaf8082e60eff5298ff4e6513edc386c5bae0274
 2020-10-12 13:42:08 +01:00
Steve Loughran
7cf5bdeec0 Revert "HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375)" 
This reverts commit 263b7d5dfced0e3d594458c664f14df64589bbc8.
 2020-10-12 10:45:18 +01:00
Akira Ajisaka
263b7d5dfc HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 10:43:26 +01:00
Doroszlai, Attila
13e0c5f6e0 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-08 23:44:56 -07:00
Jim Brennan
76e223a320 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 18:59:25 +00:00
Steve Loughran
963793dd48
HADOOP-17293. S3A to always probe S3 in S3A getFileStatus on non-auth paths 
This reverts changes in HADOOP-13230 to use S3Guard TTL in choosing when
to issue a HEAD request; fixing tests to compensate.

New org.apache.hadoop.fs.s3a.performance.OperationCost cost,
S3GUARD_NONAUTH_FILE_STATUS_PROBE for use in cost tests.

Contributed by Steve Loughran.

Change-Id: I418d55d2d2562a48b2a14ec7dee369db49b4e29e
 2020-10-08 15:38:32 +01:00