big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,333 Commits 2 Branches 2 Tags 264 MiB
35eff54556
Commit Graph

25333 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
Viraj Jasani
e95c3259de
MAPREDUCE-7356. Remove some duplicate dependencies from mapreduce-client's child poms (#3193). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-13 19:30:13 +05:30
Anoop Sam John
177d906a67
HADOOP-17770 WASB : Support disabling buffered reads in positional reads (#3149) 2021-07-13 10:37:12 +05:30
Akira Ajisaka
c81f82e21d
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
 2021-07-12 10:16:13 +09:00
litao
fef53aacc9
HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-10 13:55:11 +05:30
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Akira Ajisaka
6ac5d8a1ed
HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 2021-07-09 22:48:47 +09:00
S O'Donnell
b6c06c4b76 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 2021-07-09 10:03:57 +01:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
Eric Payne
7581413156 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 2021-07-07 20:43:44 +00:00
JiaguodongF
9381071758
HDFS-16114. the balancer parameters print error (#3179) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-07 14:13:05 +05:30
Shubham Gupta
3f4221ec34
MAPREDUCE-7351 - CleanupJob during handle of SIGTERM signal (#3176) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2021-07-07 09:08:15 +05:30
Masatake Iwasaki
c42441a1ec
HADOOP-17775. Remove JavaScript package from Docker environment. (#3137) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 11:07:10 +09:00
Ayush Saxena
c85a18327d HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w. 2021-07-06 23:26:52 +05:30
litao
a447cfd78b
HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-07-05 19:31:22 -07:00
liangxs
a5db6831bc
HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 2021-07-06 09:11:03 +08:00
Rafal Wojdyla
f639fbc29f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-05 21:07:12 +01:00
Mukund Thakur
93ad7c32f4
HADOOP-17250 Lot of short reads can be merged with readahead. (#3110) 
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The 
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur
 2021-07-05 15:49:13 +05:30
Viraj Jasani
da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 2021-07-05 09:23:24 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-05 08:14:37 +09:00
sumangala-patki
35570e414a
HADOOP-17290. ABFS: Add Identifiers to Client Request Header (#2520) 
Contributed by Sumangala Patki.
 2021-07-02 19:13:20 +05:30
Mehakmeet Singh
ea259f236c
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value (#3144) 
Contributed by: Mehakmeet Singh
 2021-07-02 14:03:16 +01:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
zhangshuyan0
9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. (#3159) 
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
 2021-07-01 23:52:21 +08:00
litao
52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-01 18:44:10 +05:30
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
Gautham B A
7c999e2d9a
HADOOP-17778. CI for Centos 8 (#3151) 2021-06-30 22:04:54 -07:00
Viraj Jasani
d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:55:07 -07:00
Akira Ajisaka
20a4b1ae36
HADOOP-17331. [JDK 16] TestDNS fails (#2884) 2021-06-30 03:06:29 -07:00
Xiangyi Zhu
4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-06-30 00:12:08 -07:00
Wanqiang Ji
c005195b78
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 2021-06-29 18:32:16 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
Viraj Jasani
93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 2021-06-29 02:31:02 -07:00
Jim Brennan
dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Prabhu Joseph
6b4987f09c YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 2021-06-27 16:28:32 +05:30
Masatake Iwasaki
3788fe52da HDFS-13916. Distcp SnapshotDiff to support WebHDFS. Contributed by Xun REN. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-06-26 21:04:56 +00:00
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Zamil Majdy
ed5d10ee48
HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt (#3109) 
Contributed by Zamil Majdy.
 2021-06-25 20:01:48 +01:00
He Xiaoqiao
748570b73c
Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." 
This reverts commit 0d07837712.
 2021-06-25 22:38:34 +08:00
Peter Bacsko
0934e783cf YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori. 2021-06-24 20:15:10 +02:00
He Xiaoqiao
0d07837712
HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun. 2021-06-25 01:09:42 +08:00
Steve Loughran
5b7f68ac76
HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain." (#3133) 
This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint
is set in fs.s3a.endpoint, S3A filesystem creation may fail on
non-EC2 deployments, depending on the local host environment setup.

* If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region
  is null, the region is set to "us-east-1".
* If fs.s3a.endpoint.region is explicitly set to "" then the client
  falls back to the SDK region resolution chain; this works on EC2
* Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+
* Also contains some minor restructuring of troubleshooting.md

Contributed by Steve Loughran.
 2021-06-24 16:37:27 +01:00
Ahmed Hussein
581f43dce1
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3130). Contributed by Ahmed Hussein. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 17:57:52 +05:30
litao
f78b6d47f9
HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 15:59:42 +05:30
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Gautham B A
ef5dbc7283
HADOOP-17766. CI for Debian 10 (#3129) 2021-06-23 10:02:33 -07:00
Masatake Iwasaki
11d144d228
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 2021-06-23 19:33:03 +09:00
Viraj Jasani
d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 2021-06-22 20:21:34 -07:00