big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Projects Wiki Activity
24,935 Commits 2 Branches 2 Tags
Commit Graph

24935 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Neil
cd5ee0014f YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 
(cherry picked from commit fa4cf91b5750327ed67355c35f1ed0f10bb0100d)
 2021-01-08 10:38:08 -08:00
Szilard Nemeth
f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
Steve Loughran
a2ae0d7079
Revert "HADOOP-17430. Restore ability to set Text to empty byte array (#2545)" 
This reverts commit 9e85eb9a2ef1ba2e4e19f233e0cc114dfd07a6dc.

Change-Id: Id1ac803b29931b0f643cb37bbe58534726c36f1e
 2021-01-08 10:50:28 +00:00
srinivasst
98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a8a45ff8c7d25d52cb014d0ce858cc2)
 2021-01-08 10:57:37 +05:30
Akira Ajisaka
3748c258f9
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 
(cherry picked from commit ae4945fb2c89397f81e7d0566784e6548b6de685)
 2021-01-06 12:14:09 +09:00
dgzdot
9e85eb9a2e HADOOP-17430. Restore ability to set Text to empty byte array (#2545) 
Contributed by gaozhan.ding

Change-Id: Ib2ad9120c15c46a3fa2de9e3206875cbbc2363c2
 2021-01-05 21:15:14 +00:00
Wei-Chiu Chuang
94c126cc9e HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
 2021-01-04 20:56:18 -08:00
Wei-Chiu Chuang
6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
 2021-01-04 11:28:26 -08:00
He Xiaoqiao
cfcd17ffe7
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:32:24 +08:00
Liang-Chi Hsieh
060594259c HDFS-15690. Add lz4-java as test dependency (#2481) 2020-12-29 13:36:04 -08:00
Gautham B A
98fe00e208 HDFS-15699 Remove lz4 references in vcxproj (#2498) 2020-12-29 13:34:54 -08:00
dengzh
bc0c3ece0f HADOOP-17390. Skip license check on lz4 code files (#2478) 
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
 2020-12-29 13:32:26 -08:00
Liang-Chi Hsieh
87064df1f2 HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-12-29 13:17:26 -08:00
Masatake Iwasaki
b8a4361d7b HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-12-29 13:11:51 -08:00
He Xiaoqiao
3736f6eb57
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:47:16 +08:00
Eric Badger
1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)
 2020-12-21 23:11:47 +00:00
yzhangal
adf6ca18b4
HADOOP-17338. Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2497) 
Yongjun Zhang <yongjunzhang@pinterest.com>

Change-Id: Ibbc6a39afb82de1208e6ed6a63ede224cc425466
 2020-12-19 12:24:16 +00:00
Ahmed Hussein
be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234256e342fd9b5ef6fe4455471009693)
 2020-12-17 17:50:47 +00:00
He Xiaoqiao
3a860e876e HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
 2020-12-15 14:01:48 -08:00
Ayush Saxena
faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
Chao Sun
81e533de8f
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. Contributed by Chao Sun (#2522) 2020-12-12 09:37:13 -08:00
S O'Donnell
1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c9c3d54f618e802fddacbafbe828211)
 2020-12-11 21:05:00 +00:00
Jim Brennan
e5f11ea5b2 HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 20:19:08 +00:00
Ahmed Hussein
16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil
bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50672ebdd7f12b9a2235aec37b2a07cf)
 2020-12-10 13:50:32 -08:00
Jim Brennan
27e455a1a0 HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 20:57:43 +00:00
Liang-Chi Hsieh
485e4e3348 HADOOP-17425. Bump up snappy-java to 1.1.8.2. (#2536) 
(cherry picked from commit c2cecfc9b951e21d38783227379bec63f34b7702)

 Conflicts:
	hadoop-project/pom.xml
 2020-12-10 13:58:49 +00:00
Masatake Iwasaki
3a50148809 HADOOP-17369. Bump up snappy-java to 1.1.8.1. (#2451) 
(cherry picked from commit 0946f58caecd47c035b1805e2a668dd5d2954dec)
 2020-12-10 13:57:35 +00:00
Mukund Thakur
e4cab4b7a3
HADOOP-17186. Fixing javadoc in ListingOperationCallbacks (#2196) 
(cherry picked from commit ac697571a13d3d18293e953ea2648bcd74b9069f)
 2020-12-10 18:32:22 +09:00
Akira Ajisaka
b84bc63aa2 HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-10 05:51:45 +05:30
Ayush Saxena
8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Ankit Kumar
f04a9dfda1
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320a9d11d2fd8c86fb5fa4e2d1c45d9ef)
 2020-12-09 10:53:42 +09:00
Ahmed Hussein
e02b179c4c HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a21722be706d7ab682323285e794f71cadf)
 2020-12-08 19:28:44 +00:00
Hui Fei
cb2dce30d4 HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 10:40:14 +08:00
crossfire
edd9b659ca HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d5765c98d41c78736124b7b7f078aa2)
 2020-12-07 15:50:38 -08:00
Jim Brennan
6a5864ee4a HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 2020-12-07 21:19:36 +00:00
Ahmed Hussein
30c9f3920f HADOOP-17389. KMS should log full UGI principal. (#2476) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
(cherry picked from commit 8c234fc0d4472b65ff555a79acb9a12f2dae3b27)
 2020-12-07 17:12:52 +00:00
Chao Sun
4628647067 HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:43:25 -08:00
Akira Ajisaka
efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 
(cherry picked from commit 3e7b65c3e5920ed78bd87b68de1075003d86e7a0)
 2020-12-04 10:07:58 +09:00
Ahmed Hussein
a7dbd3b6f9 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040806c4c9687288de89e42cfacf0365e)
 2020-12-03 15:15:18 -08:00
Jim Brennan
5bfb97bc7d HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-03 17:59:01 +00:00
Andrea Scarpino
c5b9c5dfe5
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:16:45 +09:00
Thomas Marquardt
a5695057b1
HADOOP-17397: ABFS: SAS Test updates for version and permission update 
DETAILS:

    The previous commit for HADOOP-17397 was not the correct fix.  DelegationSASGenerator.getDelegationSAS
    should return sp=p for the set-permission and set-acl operations.  The tests have also been updated as
    follows:

    1. When saoid and suoid are not specified, skoid must have an RBAC role assignment which grants
       Microsoft.Storage/storageAccounts/blobServices/containers/blobs/modifyPermissions/action and sp=p
       to set permissions or set ACL.

    2. When saoid or suiod is specified, same as 1) but furthermore the saoid or suoid must be an owner of
       the file or directory in order for the operation to succeed.

    3. When saoid or suiod is specified, the ownership check is bypassed by also including 'o' (ownership)
       in the SAS permission (for example, sp=op).  Note that 'o' grants the saoid or suoid the ability to
       change the file or directory owner to themself, and they can also change the owning group. Generally
       speaking, if a trusted authorizer would like to give a user the ability to change the permissions or
       ACL, then that user should be the file or directory owner.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 461, Failures: 0, Errors: 0, Skipped: 24
    Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 461, Failures: 0, Errors: 0, Skipped: 70
    Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
 2020-12-03 14:31:06 +00:00
Ahmed Hussein
2249edb001 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69044a0770390f674d93f57e4bab3880)
 2020-12-02 21:40:17 +00:00
Ahmed Hussein
0f18a19d97 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7f17f095548b4a4485e7b7218af74be)
 2020-12-02 20:12:31 +00:00
Eric Payne
1fd6d81617 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit fa773a832652ac6f2a24f93fc2bfc2ae251bfe1d)
 2020-12-01 22:51:20 +00:00
Ahmed Hussein
a660714387 HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82fb0bc6f96301c7ca74a32018b60c49e)
 2020-12-01 21:16:49 +00:00
kevinzhao1661
7d9c6edadf
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:17:59 +09:00
Mukund Thakur
3ef0e3d615 HADOOP-17398. Skipping network I/O in S3A getFileStatus(/) breaks some tests (#2493) 
Follow-on to HADOOP-17323.

Contributed by Mukund Thakur.
 2020-11-26 20:26:44 +00:00