big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,059 Commits 2 Branches 2 Tags
Commit Graph

24059 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ahmed Hussein
75ca0c0f23 HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit ebe1d1fbf7a7549078e5a468080513db09b6416f)
 2020-11-13 21:14:47 +00:00
Takanobu Asanuma
d78bf82722 HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d9381d13329d25bb4f367e5378b6b5b)
 2020-11-13 11:05:20 +09:00
Ahmed Hussein
23fe3bdab3 HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0fcaf73a3989000dbc60fa214a317da1)
 2020-11-12 10:35:28 -08:00
Doroszlai, Attila
47131cdf7c
HADOOP-17365. Contract test for renaming over existing file is too lenient (#2447) 
Contributed by Attila Doroszlai.

Change-Id: I21c29256b52449b7fea335704b3afa02e39c6a39
 2020-11-11 21:21:11 +00:00
Chao Sun
8e4b1cd192 HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade (#2458) 2020-11-11 09:49:20 -08:00
Chao Sun
64344fb302 HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars (#2411) 
Contributed by Chao Sun.
 2020-11-11 09:48:57 -08:00
Stephen Jung
0712505b59 HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 
(cherry picked from commit 45434c93e865386de8c5f4036e795a3362557d56)
 2020-11-10 11:41:21 -08:00
Gabriel Medeiros Coelho
9a3f2bb545 YARN-10480. replace href tags with ng-href (#2426) 
(cherry picked from commit 61f8c5767e8d72bce6fd560c332ff8fc68d05ad2)
 2020-11-10 11:00:00 -08:00
Yizhong Zhang
12bfd9194a HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794fe119946bec2e563c1937f312e8ac9)
 2020-11-10 09:14:12 -08:00
Peter Bacsko
0361837c9e YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko. 2020-11-10 13:16:52 +01:00
Akira Ajisaka
aa3807ed1d
HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. (#2433) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 34e2abd811b271708c82bf4d5bedfc9d721366bf)
 2020-11-10 14:00:08 +09:00
Steve Loughran
7cb5325dda HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 17:05:23 +05:30
hchaverr
043cca01b1 HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 
(cherry picked from commit 6eacaffeea21e7e9286497ee17f89fd939d2eead)
 2020-11-06 14:31:31 -08:00
Eric E Payne
1e22929885 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bddeeb98e63f09882420505e3cfaedad1)
 2020-11-05 22:08:51 +00:00
Eric Badger
c6fee0a2c8 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.

(cherry picked from commit af389d989770c4218c24e640f572bc7492f7d4b3)
 2020-11-05 21:56:46 +00:00
Jim Brennan
41d58d190d Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387)" 
This reverts commit e21b81276e04138301dae1aa88da7bc4a51722fd.
 2020-11-05 17:31:39 +00:00
Wei-Chiu Chuang
cfa0986d00 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit dd1634ec3b676d6c3da18af7da46734584ddd9f6.
 2020-11-04 16:18:23 -08:00
Ayush Saxena
cb11fd8793 HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:44:08 +05:30
Konstantin V Shvachko
e48dd9daea HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4aa0d6cc9f9be530802e54610776a4d)
 2020-11-03 14:52:05 -08:00
Akira Ajisaka
dd1634ec3b HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04cb8e5dab8410ef64daa0fe799e389fd)
 2020-11-03 11:22:48 -08:00
Eric E Payne
2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan
cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518bfdf7ad257cc1cf3c73f4208c49fc0)
 2020-10-29 19:52:43 +00:00
Sunil G
91a3d298b9 HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d527b0089124a8ca69b36fb20ed118c55)
 2020-10-29 07:20:46 +05:30
Jonathan Hung
f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
 2020-10-28 10:33:41 -07:00
Brahma Reddy Battula
fe8b22ca8f HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1e48db05ab827d0cf09963a430615ed)
 2020-10-28 09:28:09 -07:00
Akira Ajisaka
8e0922edb2 HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 
(cherry picked from commit b3ba74d72dfe938ea7ed9e6b0870db6c87aa0a91)
 2020-10-28 19:14:19 +09:00
Ahmed Hussein
d4129c6794
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac43a480a010d40c182c1d4cc5346db8f)
 2020-10-27 20:49:15 +09:00
Ahmed Hussein
1429c65533
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d3325e0b454475fa01cc50f68007b0e15c)
 2020-10-27 16:58:04 +09:00
Ayush Saxena
af5f90623c HADOOP-17328. LazyPersist Overwrite fails in direct write mode. (#2413) 
(cherry picked from commit 872440610f066d4b12c9f93c05477848a260b21f)
 2020-10-27 01:40:25 +09:00
Hui Fei
b8cc09d7c7 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827c80ea37ed5207a65831d0b699c8381)
 2020-10-26 22:25:03 +08:00
Anoop Sam John
8312f230eb HADOOP-17308. WASB PageBlobOutputStream.flush succeeds even when flush to storage fails (#2392) 
Contributed by Anoop Sam John.
 2020-10-26 13:31:53 +00:00
Wei-Chiu Chuang
c8eb86ee3d HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002717a9684d00f0e9dc263ab7eb49246)
 2020-10-23 12:06:23 -07:00
Inigo Goiri
c55d639e8f HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a26988fcf5880311067e62a15e05fc1e)
 2020-10-23 09:48:23 -07:00
Vinayakumar B
e21b81276e
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:34:14 +05:30
Kihwal Lee
02709cb054 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc286db00b385f3280627d2b2063b4e59)
 2020-10-22 19:41:38 -05:00
Akira Ajisaka
c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko. 
(cherry picked from commit 7bc305db5d5a50127dd7780534d6d7d0b7c683d1)
 2020-10-23 04:41:37 +09:00
Kihwal Lee
cf932a7e2d HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-22 09:55:28 -05:00
Takanobu Asanuma
0bb1f0df27 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:12:26 +09:00
tkuramoto33
0289cdc43c
HADOOP-17298. Backslash in username causes build failure in the environment started by start-build-env.sh. (#2367) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a2e7a073f03f682d6716ed59c60533be09841784)
 2020-10-20 10:11:18 +09:00
Akira Ajisaka
37c2e34b02
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb4207af2f06f8ef57e83ff9759871d0cb)
 2020-10-19 10:49:21 +09:00
Ayush Saxena
54c40cbf49
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same (#2383) 
Contributed by Gabor Bota.
 2020-10-17 01:34:01 +05:30
Kihwal Lee
433434d509 HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:16:47 -05:00
Eric Badger
8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780c1505ab7f302c3116a0c5cdd122315)
 2020-10-15 18:09:00 +00:00
Sneha Vijayarajan
d5b4d04b0d
HADOOP-17301. ABFS: read-ahead error reporting breaks buffer management (#2369) 
Fixes read-ahead buffer management issues introduced by HADOOP-16852,
 "ABFS: Send error back to client for Read Ahead request failure".

Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
da5db6a5a6
HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. 
Contributed by Sneha Vijayarajan

Testing:

namespace.enabled=false
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 246
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 33
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
d166420302
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-10-14 22:29:13 +00:00
bilaharith
f208da286c
HADOOP-17166. ABFS: configure output stream thread pool (#2179) 
Adds the options to control the size of the per-output-stream threadpool
when writing data through the abfs connector

* fs.azure.write.max.concurrent.requests
* fs.azure.write.max.requests.to.queue

Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
bilaharith
cc7350302f
HADOOP-16915. ABFS: Ignoring the test ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00
Sneha Vijayarajan
4072323de4
Upgrade store REST API version to 2019-12-12 
- Contributed by Sneha Vijayarajan
 2020-10-14 22:29:13 +00:00
bilaharith
e481d0108a
HADOOP-17149. ABFS: Fixing the testcase ITestGetNameSpaceEnabled 
- Contributed by Bilahari T H
 2020-10-14 22:29:13 +00:00