big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,512 Commits 2 Branches 2 Tags 264 MiB
05c9c2ed02
Commit Graph

24512 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Steve Loughran
05c9c2ed02 Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)" 
This reverts commit d3014e01f3.
(fixing commit text before it is frozen)
 2021-01-12 17:29:59 +00:00
Steve Loughran
d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. (#2594) 
Caused by HADOOP-16380 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.
 2021-01-12 17:25:14 +00:00
Mehakmeet Singh
0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604) 
Contributed by Mehakmeet Singh.
 2021-01-12 15:48:09 +00:00
Yiqun Lin
0d7ac54510 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 2021-01-12 23:38:55 +08:00
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
bilaharith
612330661b
HADOOP-17459. ADLS Gen1: Fixes for rename contract tests #2607 
Contributed by Bilaharith
 2021-01-12 14:00:48 +00:00
lzx404243
ca7dd5fad3
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:45:13 +09:00
Ahmed Hussein
ae894ecfcc
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:34:06 +09:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
stack
d4fd675a95 Revert "HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470)" 
This reverts commit e306f59421.
 2021-01-11 08:54:55 -08:00
Brahma Reddy Battula
1ec5c67b5e HADOOP-16492. Support HuaweiCloud Object Storage as a Hadoop Backend File System. Contributed by zhongjun. 2021-01-11 18:12:12 +05:30
He Xiaoqiao
ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein
890f2da624
YARN-10553. Refactor TestDistributedShell (#2581) 2021-01-09 08:39:21 +09:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
Borislav Iordanov
e306f59421
HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470) 
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
Signed-off-by: stack <stack@apache.org>
 2021-01-08 09:10:21 -08:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
Ayush Saxena
2ba7ec2b48 HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun. 2021-01-08 16:27:54 +05:30
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Sneha Vijayarajan
b612c310c2
HADOOP-17404. ABFS: Small write - Merge append and flush 
- Contributed by Sneha Vijayarajan
 2021-01-06 10:43:37 -08:00
bilaharith
d21c1c6576
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2551) 
Contributed by bilaharith
 2021-01-06 14:32:13 +00:00
Akira Ajisaka
ae4945fb2c
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 2021-01-06 12:12:46 +09:00
dgzdot
b1abb10ea2
HADOOP-17430. Restore ability to set Text to empty byte array (#2545) 
Contributed by gaozhan.ding
 2021-01-05 21:09:41 +00:00
Gabor Bota
42eb9ff68e
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593) 
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
 2021-01-05 15:43:01 +01:00
Wei-Chiu Chuang
2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 2021-01-04 20:54:44 -08:00
Masatake Iwasaki
5abeecd33f
YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586) 2021-01-05 13:38:34 +09:00
Ayush Saxena
77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 2021-01-04 09:43:58 -08:00
Masatake Iwasaki
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. (#2501) 2021-01-04 14:00:57 +09:00
He Xiaoqiao
c98d274db1
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:25:17 +08:00
bilaharith
1448add08f
HADOOP-17347. ABFS: Read optimizations 
- Contributed by Bilahari T H
 2021-01-02 10:37:10 -08:00
Sneha Vijayarajan
5ca1ea89b3
HADOOP-17407. ABFS: Fix NPE on delete idempotency flow 
- Contributed by Sneha Vijayarajan
 2021-01-02 10:22:10 -08:00
Ayush Saxena
1cd96e8dd8 HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. 2021-01-01 11:37:52 +05:30
Steve Loughran
617af28e80
HADOOP-17271. S3A connector to support IOStatistics. (#2580) 
S3A connector to support the IOStatistics API of HADOOP-16830,

This is a major rework of the S3A Statistics collection to

* Embrace the IOStatistics APIs
* Move from direct references of S3AInstrumention statistics
  collectors to interface/implementation classes in new packages.
* Ubiquitous support of IOStatistics, including:
  S3AFileSystem, input and output streams, RemoteIterator instances
  provided in list calls.
* Adoption of new statistic names from hadoop-common

Regarding statistic collection, as well as all existing
statistics, the connector now records min/max/mean durations
of HTTP GET and HEAD requests, and those of LIST operations.

Contributed by Steve Loughran.
 2020-12-31 21:55:39 +00:00
Steve Loughran
9b2956e254
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579) 
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null. 

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.
 2020-12-31 16:02:10 +00:00
Steve Loughran
99d08a19ba
HADOOP-17450. Add Public IOStatistics API. (#2577) 
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.

New Packages

org.apache.hadoop.fs.statistics. 
  Public API, including:
    IOStatisticsSource
    IOStatistics
    IOStatisticsSnapshot (seralizable to java objects and json)
    +helper classes for logging and integration
    BufferedIOStatisticsInputStream
       implements IOStatisticsSource and StreamCapabilities
     BufferedIOStatisticsOutputStream
       implements IOStatisticsSource, Syncable and StreamCapabilities
       

org.apache.hadoop.fs.statistics.impl
  Implementation classes for internal use.

org.apache.hadoop.util.functional
  functional programming support for RemoteIterators and
  other operations which raise IOEs; all wrapper classes
  implement and propagate IOStatisticsSource
    
Contributed by Steve Loughran.
 2020-12-31 11:52:42 +00:00
Neil
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Ahmed Hussein
513f1995ad
YARN-10334. Close clients in TestDistributedShell (#2571) 2020-12-28 11:09:10 -08:00
maobaolong
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable (#2572) 2020-12-28 14:18:18 +08:00
He Xiaoqiao
005b854f6b
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:44:12 +08:00
hemanthboyina
16a20503ca HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 2020-12-22 17:27:31 +05:30
langlaile1221
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566) 2020-12-22 09:25:11 +08:00
Eric Badger
4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.
 2020-12-21 23:10:31 +00:00
Masatake Iwasaki
9aefe72ed6
HDFS-15743. Fix -Pdist build failure of hadoop-hdfs-native-client. (#2569) 2020-12-22 07:19:44 +09:00
Ayush Saxena
ba496f61f8 HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina. 2020-12-22 02:16:00 +05:30
Sumangala
a35fc3871b
HADOOP-17422: ABFS: Set default ListMaxResults to max server limit (#2535) 
Contributed by Sumangala Patki

TEST RESULTS:

namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
 2020-12-21 06:40:36 +00:00
Ayush Saxena
5ab1db9e21 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 2020-12-21 09:58:30 +05:30