big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,192 Commits 2 Branches 2 Tags 264 MiB
16e6030e25
Commit Graph

25192 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kihwal Lee
47002719f2 HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 2021-07-16 14:22:39 -05:00
Mehakmeet Singh
997d749f8a
HADOOP-17801. No error message reported when bucket doesn't exist in S3AFS (#3202) 
Contributed by: Mehakmeet Singh.
 2021-07-16 15:27:00 +01:00
Mehakmeet Singh
f6f105c7de
HADOOP-17803. Remove WARN logging from LoggingAuditor when executing a request outside an audit span (#3207) 
Followup to HADOOP-17511. "Add audit/telemetry logging to S3A connector"

Contributed by Mehakmeet Singh
 2021-07-16 11:47:05 +01:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 (#3204) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-16 12:17:17 +09:00
crossfire
4bb25c810b
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 2021-07-15 19:58:12 -07:00
Jim Brennan
632f64cadb YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 14:23:31 +00:00
Viraj Jasani
df44178eb6
HADOOP-17795. Provide fallbacks for callqueue.impl and scheduler.impl (#3192) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-14 20:58:32 +09:00
jianghuazhu
87e0000137
HADOOP-17672.Remove an invalid comment content in the FileContext class. (#2961) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-14 00:15:02 -07:00
Abhishek Das
1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
Viraj Jasani
e95c3259de
MAPREDUCE-7356. Remove some duplicate dependencies from mapreduce-client's child poms (#3193). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-13 19:30:13 +05:30
Anoop Sam John
177d906a67
HADOOP-17770 WASB : Support disabling buffered reads in positional reads (#3149) 2021-07-13 10:37:12 +05:30
Akira Ajisaka
c81f82e21d
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
 2021-07-12 10:16:13 +09:00
litao
fef53aacc9
HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-10 13:55:11 +05:30
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Akira Ajisaka
6ac5d8a1ed
HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 2021-07-09 22:48:47 +09:00
S O'Donnell
b6c06c4b76 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 2021-07-09 10:03:57 +01:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
Eric Payne
7581413156 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 2021-07-07 20:43:44 +00:00
JiaguodongF
9381071758
HDFS-16114. the balancer parameters print error (#3179) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-07 14:13:05 +05:30
Shubham Gupta
3f4221ec34
MAPREDUCE-7351 - CleanupJob during handle of SIGTERM signal (#3176) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2021-07-07 09:08:15 +05:30
Masatake Iwasaki
c42441a1ec
HADOOP-17775. Remove JavaScript package from Docker environment. (#3137) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 11:07:10 +09:00
Ayush Saxena
c85a18327d HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w. 2021-07-06 23:26:52 +05:30
litao
a447cfd78b
HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-07-05 19:31:22 -07:00
liangxs
a5db6831bc
HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 2021-07-06 09:11:03 +08:00
Rafal Wojdyla
f639fbc29f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-05 21:07:12 +01:00
Mukund Thakur
93ad7c32f4
HADOOP-17250 Lot of short reads can be merged with readahead. (#3110) 
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The 
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur
 2021-07-05 15:49:13 +05:30
Viraj Jasani
da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 2021-07-05 09:23:24 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-05 08:14:37 +09:00
sumangala-patki
35570e414a
HADOOP-17290. ABFS: Add Identifiers to Client Request Header (#2520) 
Contributed by Sumangala Patki.
 2021-07-02 19:13:20 +05:30
Mehakmeet Singh
ea259f236c
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value (#3144) 
Contributed by: Mehakmeet Singh
 2021-07-02 14:03:16 +01:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
zhangshuyan0
9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. (#3159) 
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
 2021-07-01 23:52:21 +08:00
litao
52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-01 18:44:10 +05:30
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
Gautham B A
7c999e2d9a
HADOOP-17778. CI for Centos 8 (#3151) 2021-06-30 22:04:54 -07:00
Viraj Jasani
d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:55:07 -07:00
Akira Ajisaka
20a4b1ae36
HADOOP-17331. [JDK 16] TestDNS fails (#2884) 2021-06-30 03:06:29 -07:00
Xiangyi Zhu
4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-06-30 00:12:08 -07:00
Wanqiang Ji
c005195b78
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 2021-06-29 18:32:16 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
Viraj Jasani
93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 2021-06-29 02:31:02 -07:00
Jim Brennan
dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Prabhu Joseph
6b4987f09c YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 2021-06-27 16:28:32 +05:30
Masatake Iwasaki
3788fe52da HDFS-13916. Distcp SnapshotDiff to support WebHDFS. Contributed by Xun REN. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-06-26 21:04:56 +00:00
Jim Brennan
7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Zamil Majdy
ed5d10ee48
HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt (#3109) 
Contributed by Zamil Majdy.
 2021-06-25 20:01:48 +01:00
He Xiaoqiao
748570b73c
Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." 
This reverts commit 0d07837712.
 2021-06-25 22:38:34 +08:00