big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,733 Commits 2 Branches 2 Tags 264 MiB
c8d327a4f1
Commit Graph

24733 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt
8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Akira Ajisaka
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 2021-01-21 10:07:46 +09:00
Szilard Nemeth
cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G
4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Steve Loughran
68bc721841
HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole. (#2600) 
Contributed by Steve Loughran.
 2021-01-19 17:19:27 +00:00
Szilard Nemeth
a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth
6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Ayush Saxena
9b4f09a6bb HDFS-15758. Fix typos in MutableMetric. Contributed by Haibin Huang. 2021-01-18 20:35:37 +05:30
Szilard Nemeth
acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
He Xiaoqiao
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Wangda Tan
3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori) 
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
 2021-01-15 14:20:08 -08:00
Szilard Nemeth
6cd540e964 YARN-7200. SLS generates a realtimetrack.json file but that file is missing the closing ']'. Contributed by Agshin Kazimli 2021-01-15 22:32:30 +01:00
Masatake Iwasaki
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-14 16:19:35 +09:00
Eric Badger
768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne
d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth
836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt
61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
touchida
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-13 11:23:07 +09:00
Steve Loughran
724edb0354
HADOOP-17451. IOStatistics test failures in S3A code. (#2594) 
Caused by HADOOP-16830 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.

Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
 2021-01-12 17:30:32 +00:00
Steve Loughran
05c9c2ed02 Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)" 
This reverts commit d3014e01f3.
(fixing commit text before it is frozen)
 2021-01-12 17:29:59 +00:00
Steve Loughran
d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. (#2594) 
Caused by HADOOP-16380 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.
 2021-01-12 17:25:14 +00:00
Mehakmeet Singh
0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604) 
Contributed by Mehakmeet Singh.
 2021-01-12 15:48:09 +00:00
Yiqun Lin
0d7ac54510 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 2021-01-12 23:38:55 +08:00
Szilard Nemeth
85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth
9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
bilaharith
612330661b
HADOOP-17459. ADLS Gen1: Fixes for rename contract tests #2607 
Contributed by Bilaharith
 2021-01-12 14:00:48 +00:00
lzx404243
ca7dd5fad3
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:45:13 +09:00
Ahmed Hussein
ae894ecfcc
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-12 15:34:06 +09:00
Wangda Tan
b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori) 
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
 2021-01-11 17:46:09 -08:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
stack
d4fd675a95 Revert "HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470)" 
This reverts commit e306f59421.
 2021-01-11 08:54:55 -08:00
Brahma Reddy Battula
1ec5c67b5e HADOOP-16492. Support HuaweiCloud Object Storage as a Hadoop Backend File System. Contributed by zhongjun. 2021-01-11 18:12:12 +05:30
He Xiaoqiao
ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein
890f2da624
YARN-10553. Refactor TestDistributedShell (#2581) 2021-01-09 08:39:21 +09:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
Borislav Iordanov
e306f59421
HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470) 
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
Signed-off-by: stack <stack@apache.org>
 2021-01-08 09:10:21 -08:00
Szilard Nemeth
4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth
580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
Ayush Saxena
2ba7ec2b48 HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun. 2021-01-08 16:27:54 +05:30
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
srinivasst
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T
 2021-01-08 10:52:52 +05:30
Sneha Vijayarajan
b612c310c2
HADOOP-17404. ABFS: Small write - Merge append and flush 
- Contributed by Sneha Vijayarajan
 2021-01-06 10:43:37 -08:00
bilaharith
d21c1c6576
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2551) 
Contributed by bilaharith
 2021-01-06 14:32:13 +00:00
Akira Ajisaka
ae4945fb2c
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 2021-01-06 12:12:46 +09:00
dgzdot
b1abb10ea2
HADOOP-17430. Restore ability to set Text to empty byte array (#2545) 
Contributed by gaozhan.ding
 2021-01-05 21:09:41 +00:00
Gabor Bota
42eb9ff68e
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593) 
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
 2021-01-05 15:43:01 +01:00
Wei-Chiu Chuang
2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 2021-01-04 20:54:44 -08:00
Masatake Iwasaki
5abeecd33f
YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586) 2021-01-05 13:38:34 +09:00