Commit Graph

5164 Commits

Author SHA1 Message Date
LeonGao
b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284)
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
2021-08-25 17:40:12 -07:00
Viraj Jasani
1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329)
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-08-25 17:37:25 +09:00
Viraj Jasani
6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-16 14:54:25 +09:00
wangzhaohui
e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270)
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-08-06 18:23:00 +08:00
zhangshuyan0
10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang.
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-08-04 00:22:21 +08:00
Wei-Chiu Chuang
6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-30 08:29:56 -07:00
S O'Donnell
4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
zhihaixu2012
ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175)
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
2021-07-27 19:18:44 -07:00
JiaguodongF
aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185)
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-07-27 22:25:21 +05:30
Viraj Jasani
b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-27 14:57:12 +08:00
Akira Ajisaka
d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229)
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-07-26 16:46:07 +09:00
litao
5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-07-26 12:45:01 +08:00
He Xiaoqiao
6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00
Abhishek Das
1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao
ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639)
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
2021-07-13 14:18:59 -07:00
bshashikant
6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171)
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
litao
b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
litao
b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172)
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-05 08:14:37 +09:00
litao
390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
litao
56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
litao
95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115)
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-06-21 10:25:12 +09:00
Ahmed Hussein
c255feb5d7
HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein.
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:22:47 +05:30
LeonGao
643dfd60e4
HDFS-15842. HDFS mover to emit metrics. (#2738) 2021-06-19 15:39:46 -07:00
hust_hhb
56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065) 2021-06-17 11:08:29 +08:00
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998)
Contributed by Viraj Jasani
2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
Ahmed Hussein
352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097) 2021-06-11 11:00:42 -05:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084)
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-06-11 17:30:55 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036)
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)"
This reverts commit 041488e8f5.
2021-06-01 08:38:13 +05:30
Konstantin V Shvachko
8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko
1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 10:55:58 +09:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-05-16 00:45:11 +05:30
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-11 12:38:15 +09:00
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980)
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
2021-05-08 00:24:06 +09:00
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)
Reviewed-by: Siyao Meng <siyao@apache.org>
2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945)
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955)
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-26 18:38:43 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)"
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
2021-04-26 11:27:15 +08:00