big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,164 Commits 2 Branches 2 Tags 264 MiB
4865589bb4
Commit Graph

24164 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sneha Vijayarajan
4865589bb4 HADOOP-17404. ABFS: Small write - Merge append and flush 
- Contributed by Sneha Vijayarajan

(cherry picked from commit b612c310c2)
 2021-01-22 10:48:04 +00:00
bilaharith
cb6729224e HADOOP-17347. ABFS: Read optimizations 
- Contributed by Bilahari T H

(cherry picked from commit 1448add08f)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
f3a0ca66c2 HADOOP-17407. ABFS: Fix NPE on delete idempotency flow 
- Contributed by Sneha Vijayarajan

(cherry picked from commit 5ca1ea89b3)
 2021-01-22 10:48:04 +00:00
Sumangala
5f312a0d85 HADOOP-17422: ABFS: Set default ListMaxResults to max server limit (#2535) 
Contributed by Sumangala Patki

TEST RESULTS:

namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141

(cherry picked from commit a35fc3871b)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
d3caa1552b Hadoop-17413. Release elastic byte buffer pool at close 
- Contributed by Sneha Vijayarajan

(cherry picked from commit 5bf977e6b1)
 2021-01-22 10:48:04 +00:00
Sneha Vijayarajan
a44890eb63 HADOOP-17296. ABFS: Force reads to be always of buffer size. 
Contributed by Sneha Vijayarajan.

(cherry picked from commit 142941b96e)
 2021-01-22 10:48:04 +00:00
bibinchundatt
1520b84b36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 
(cherry picked from commit 8bc2dfbf36)
 2021-01-22 08:30:12 +05:30
Maksim Bober
763157dd12
HADOOP-17484. Typo in hadop-aws index.md (#2634) 
Contributed by Maksim Bober.

Change-Id: Ic5196a64abc68566a3542e9ff96042593f081bdd
 2021-01-21 17:32:03 +00:00
Akira Ajisaka
03c84fb9be
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:09:17 +09:00
Steve Loughran
b645e58de2
HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole. (#2600) 
Contributed by Steve Loughran.

Change-Id: Iece617be78e80fc7e956074eddf171f7763a2e66
 2021-01-19 17:20:28 +00:00
He Xiaoqiao
26cd02fb29
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-19 23:29:45 +08:00
Steve Loughran
56576f080b
HADOOP-17451. IOStatistics test failures in S3A code. (#2594) 
Caused by HADOOP-16830 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.

Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
 2021-01-14 13:21:20 +00:00
Steve Loughran
240b25310e
HADOOP-17271. S3A connector to support IOStatistics. (#2580) 
S3A connector to support the IOStatistics API of HADOOP-16830,

This is a major rework of the S3A Statistics collection to

* Embrace the IOStatistics APIs
* Move from direct references of S3AInstrumention statistics
  collectors to interface/implementation classes in new packages.
* Ubiquitous support of IOStatistics, including:
  S3AFileSystem, input and output streams, RemoteIterator instances
  provided in list calls.
* Adoption of new statistic names from hadoop-common

Regarding statistic collection, as well as all existing
statistics, the connector now records min/max/mean durations
of HTTP GET and HEAD requests, and those of LIST operations.

Contributed by Steve Loughran.

Change-Id: I182d34b6ac39e017a8b4a221dad8e930882b39cf
 2021-01-14 13:21:01 +00:00
Steve Loughran
5be450393c
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579) 
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.

Change-Id: Iff1485297c2c7e181b54eaf1d2c4f80faeee7cfa
 2021-01-14 13:20:38 +00:00
Steve Loughran
57abfae136
HADOOP-17450. Add Public IOStatistics API. (#2577) 
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics.
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource

Contributed by Steve Loughran.

Change-Id: If56e8db2981613ff689c39239135e44feb25f78e
 2021-01-14 13:20:17 +00:00
Masatake Iwasaki
89452682a5
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:20:49 +09:00
Eric Badger
7fc624aabc YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba)
 2021-01-13 23:43:15 +00:00
Eric Payne
4518ece4c9 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit d8fadec944)
 2021-01-13 17:17:07 +00:00
touchida
75f34a156f
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:25:10 +09:00
bilaharith
8204ad9d5b HADOOP-17459. ADLS Gen1: Fixes for rename contract tests #2607 
Contributed by Bilaharith
 2021-01-12 14:04:37 +00:00
lzx404243
b3f85576e9
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3)
 2021-01-12 15:45:54 +09:00
Ahmed Hussein
c6b1507eb2
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae894ecfcc)
 2021-01-12 15:35:59 +09:00
stack
b74d642220 Revert "HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470)" 
This reverts commit f7d2a5d7a52c41cba14b17eb0c9189d983f202cf.
 2021-01-11 08:56:24 -08:00
Ahmed Hussein
98690134d5 YARN-10334. Close clients in TestDistributedShell (#2571) 
(cherry picked from commit 513f1995ad)
 2021-01-09 21:18:00 +09:00
Ahmed Hussein
a40eabd717 YARN-10536. Client in distributedShell swallows interrupt exceptions (#2554) 
(cherry picked from commit 7a88f45366)
 2021-01-09 21:18:00 +09:00
Masatake Iwasaki
35740a25d9 YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586) 
(cherry picked from commit 5abeecd33f)
 2021-01-09 21:18:00 +09:00
He Xiaoqiao
e95ee67632
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:10 +08:00
Michael Stack
f046ed27d6
HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470) (#2609) 
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
Signed-off-by: stack <stack@apache.org>

Co-authored-by: Borislav Iordanov <borislav.iordanov@gmail.com>
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
 2021-01-08 13:45:44 -08:00
Ahmed Hussein
18e2835766 HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 
(cherry picked from commit 77435a025e)
 2021-01-08 19:29:14 +00:00
Neil
cd5ee0014f YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 
(cherry picked from commit fa4cf91b57)
 2021-01-08 10:38:08 -08:00
Szilard Nemeth
f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
Steve Loughran
a2ae0d7079
Revert "HADOOP-17430. Restore ability to set Text to empty byte array (#2545)" 
This reverts commit 9e85eb9a2e.

Change-Id: Id1ac803b29931b0f643cb37bbe58534726c36f1e
 2021-01-08 10:50:28 +00:00
srinivasst
98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
 2021-01-08 10:57:37 +05:30
Akira Ajisaka
3748c258f9
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 
(cherry picked from commit ae4945fb2c)
 2021-01-06 12:14:09 +09:00
dgzdot
9e85eb9a2e HADOOP-17430. Restore ability to set Text to empty byte array (#2545) 
Contributed by gaozhan.ding

Change-Id: Ib2ad9120c15c46a3fa2de9e3206875cbbc2363c2
 2021-01-05 21:15:14 +00:00
Wei-Chiu Chuang
94c126cc9e HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
 2021-01-04 20:56:18 -08:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0)
 2021-01-04 11:28:26 -08:00
He Xiaoqiao
cfcd17ffe7
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:32:24 +08:00
Liang-Chi Hsieh
060594259c HDFS-15690. Add lz4-java as test dependency (#2481) 2020-12-29 13:36:04 -08:00
Gautham B A
98fe00e208 HDFS-15699 Remove lz4 references in vcxproj (#2498) 2020-12-29 13:34:54 -08:00
dengzh
bc0c3ece0f HADOOP-17390. Skip license check on lz4 code files (#2478) 
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
 2020-12-29 13:32:26 -08:00
Liang-Chi Hsieh
87064df1f2 HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-12-29 13:17:26 -08:00
Masatake Iwasaki
b8a4361d7b HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-12-29 13:11:51 -08:00
He Xiaoqiao
3736f6eb57
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:47:16 +08:00
Eric Badger
1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
 2020-12-21 23:11:47 +00:00
yzhangal
adf6ca18b4
HADOOP-17338. Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2497) 
Yongjun Zhang <yongjunzhang@pinterest.com>

Change-Id: Ibbc6a39afb82de1208e6ed6a63ede224cc425466
 2020-12-19 12:24:16 +00:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 17:50:47 +00:00
He Xiaoqiao
3a860e876e HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc00)
 2020-12-15 14:01:48 -08:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
Chao Sun
81e533de8f
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. Contributed by Chao Sun (#2522) 2020-12-12 09:37:13 -08:00