hadoop

Author	SHA1	Message	Date
Wei-Chiu Chuang	87632bbacf	HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468 )	2021-09-24 08:35:23 -07:00
Ayush Saxena	98e2781dde	HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov.	2021-09-23 10:45:02 +05:30
LeonGao	b1431813d0	HDFS-10648. Expose Balancer metrics through Metrics2 (#3427 )	2021-09-21 12:26:17 -07:00
Viraj Jasani	feee41aa00	HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-09-21 10:11:20 +09:00
Thomas Leplus	f92c6750ec	HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455 ). Contributed by Thomas Leplus.	2021-09-20 00:09:05 +05:30
Viraj Jasani	c9763a99c1	HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429 )	2021-09-18 19:02:45 +08:00
jianghuazhu	cd5c6395e8	HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451 ). Contributed by JiangHua Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-18 11:06:51 +05:30
LeonGao	deb52cdcbb	HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425 )	2021-09-16 23:59:36 -07:00
litao	2d479309cc	HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366 ) Reviewed-by: Hui Fei <ferhui@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-09-16 10:00:02 +09:00
wangzhaohui	3ecaa39668	HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317 ) Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>	2021-09-15 09:56:51 -07:00
jianghuazhu	1032104b9a	HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831 ). Contributed by JiangHua Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-14 14:24:36 +05:30
EungsopYoo	c4c5883d8b	HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359 ) Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-09-14 13:18:15 +08:00
Ayush Saxena	d9eb5ad6d3	HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404 ). Contributed by Ayush Saxena.	2021-09-13 22:50:39 +05:30
Ayush Saxena	17c58ac959	HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424 ). Contributed by Ayush Saxena.	2021-09-13 17:54:38 +08:00
bshashikant	356ebbbe80	HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340 )	2021-09-13 10:25:19 +05:30
jianghuazhu	edfde6eebc	HDFS-16197.Simplify getting NNStorage in FSNamesystem. (#3357 )	2021-09-12 19:58:30 +08:00
Symious	7f6553af75	HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421 )	2021-09-12 19:52:47 +08:00
LeonGao	90bc688c78	HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346 ) Contributed by Leon Gao * Router to support resolving monitored namenodes with DNS * Style * fix style and test failure * Add test for NNHAServiceTarget const * Resolve comments * Fix test * Comments and style * Create a simple function to extract port * Use LambdaTestUtils.intercept * fix javadoc * Trigger Build	2021-09-10 16:40:08 -07:00
Ahmed Hussein	bddc9bf63c	HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375 )	2021-09-08 21:21:16 -07:00
litao	5a306955a4	HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378 )	2021-09-08 12:38:55 +08:00
jianghuazhu	4e4d4fcc7c	HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368 )	2021-09-06 18:06:12 +08:00
Renukaprasad C	dd6b2cebed	HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244 ). Contributed by Renukaprasad C.	2021-09-05 17:07:59 +08:00
litao	364a1fbfb3	HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354 ). Contributed by tomscut.	2021-09-05 14:19:01 +08:00
langlaile1221	24af52efab	HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894 )	2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G	164608b546	HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353 ) Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2021-08-31 12:25:03 +08:00
Viraj Jasani	aa9cdf2af6	HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-26 00:37:38 -07:00
LeonGao	b53cae0ffb	HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284 ) * Add DNS resolution for QJM * Add log * Resolve comments * checkstyle * typo	2021-08-25 17:40:12 -07:00
Viraj Jasani	1b9927afe1	HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-08-25 17:37:25 +09:00
Neil	9084c728eb	HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315 )	2021-08-24 11:15:47 +08:00
litao	07627ef19e	HDFS-16177. Bug fix for Util#receiveFile (#3310 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-08-19 12:30:38 +08:00
litao	0e922aab41	HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291 )	2021-08-19 09:51:00 +08:00
jianghuazhu	53a2c65694	HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-17 16:47:10 +08:00
Viraj Jasani	6342d5e523	HDFS-16171. De-flake testDecommissionStatus (#3280 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-16 14:54:25 +09:00
Viraj Jasani	919a99bbb4	HDFS-16163. Avoid locking entire blockPinningFailures map (#3296 )	2021-08-16 13:40:05 +08:00
wangzhaohui	e85c44657c	HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270 ) Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-06 18:23:00 +08:00
wangzhaohui	a73b64f86b	HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269 ). Contributed by wangzhaohui.	2021-08-06 09:27:05 +01:00
zhangshuyan0	10a2526b0b	HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247 ) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-04 00:22:21 +08:00
Viraj Jasani	ccfa072dc7	HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang	6d77f3b6cd	HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-30 08:29:56 -07:00
jianghuazhu	6f730fd25c	HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: cxorm <lianp964@gmail.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-07-28 21:57:28 -07:00
He Xiaoqiao	683feaa1d4	HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>	2021-07-29 11:19:17 +08:00
S O'Donnell	4eae284827	HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell	2021-07-28 14:49:23 +01:00
zhihaixu2012	ae20516ebc	HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175 ) Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a Co-authored-by: Zhihai Xu <zxu@apache.org>	2021-07-27 19:18:44 -07:00
JiaguodongF	aecfcf165f	HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185 ) * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-27 22:25:21 +05:30
Viraj Jasani	b038042ece	HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228 ). Contributed by Viraj Jasani. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-27 14:57:12 +08:00
Akira Ajisaka	d710ec8d85	HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-07-26 16:46:07 +09:00
litao	5d765497c5	HDFS-16131. Show storage type for failed volumes on namenode web (#3211 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-26 12:45:01 +08:00
Akira Ajisaka	2f2f822488	HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227 ) Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin" This reverts commit `b6d839a60c`. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml	2021-07-26 01:53:39 +09:00
He Xiaoqiao	6ed7670a93	HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.	2021-07-17 21:57:00 +08:00
crossfire	4bb25c810b	HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414 )	2021-07-15 19:58:12 -07:00
Abhishek Das	1dd03cc4b5	HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260 )	2021-07-13 18:11:50 -07:00
LeonGao	ea90c5117d	HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639 ) * Rebase trunk * Fix to use FQDN and update config name * Fix javac * Style and trigger build * Trigger Build after force push * Trigger Build * Fix config names	2021-07-13 14:18:59 -07:00
Artem Smotrakov	ba325a8ada	HADOOP-17793. Better token validation (#3189 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-10 13:42:31 +09:00
S O'Donnell	b6c06c4b76	HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma	2021-07-09 10:03:57 +01:00
bshashikant	6450c1ea9a	HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188 )	2021-07-09 10:26:42 +05:30
Viraj Jasani	618c9218ee	HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171 ) Reviewed-by: Steve Loughran <stevel@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-07-08 16:03:40 +09:00
litao	b4c2647d0d	HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140 )	2021-07-08 14:10:45 +08:00
JiaguodongF	9381071758	HDFS-16114. the balancer parameters print error (#3179 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-07 14:13:05 +05:30
Ayush Saxena	c85a18327d	HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w.	2021-07-06 23:26:52 +05:30
Viraj Jasani	da575a53cf	HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169 )	2021-07-05 09:23:24 +08:00
litao	b87bac13e4	HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-05 08:14:37 +09:00
litao	390f8603d3	HDFS-16106. Fix flaky unit test TestDFSShell (#3168 )	2021-07-02 15:31:25 +08:00
zhangshuyan0	9be17339eb	HDFS-16099. Make bpServiceToActive to be volatile. (#3159 ) Co-authored-by: zhangshuyan <zqingchai@gmail.com>	2021-07-01 23:52:21 +08:00
litao	52b9319e57	HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-01 18:44:10 +05:30
litao	56c7ada7a5	HDFS-16086. Add volume information to datanode log for tracing (#3136 )	2021-06-30 22:06:56 -07:00
Viraj Jasani	d2e7003400	HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148 ) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: stack Chuang <stack@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-30 12:55:07 -07:00
Xiangyi Zhu	4cac6ec405	HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156 ) Co-authored-by: zhuxiangyi <43412979@qq.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-06-30 00:12:08 -07:00
litao	95454d821c	HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146 )	2021-06-29 03:15:12 -07:00
Viraj Jasani	93a1685073	HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150 )	2021-06-29 02:31:02 -07:00
litao	f78b6d47f9	HDFS-16085. Move the getPermissionChecker out of the read lock (#3134 ). Contributed by tomscut. Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-24 15:59:42 +05:30
litao	fdef2b4cca	HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117 )	2021-06-24 11:27:40 +09:00
Viraj Jasani	d9fbb3c508	HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127 )	2021-06-22 20:21:34 -07:00
Viraj Jasani	c488abbc79	HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-06-21 10:25:12 +09:00
litao	bdc9c8809e	HDFS-16079. Improve the block state change log (#3120 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 17:23:11 +05:30
Ahmed Hussein	c255feb5d7	HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095 ). Contributed by Ahmed Hussein. Reviewed-by: Jim Brennan <jbrennan@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:22:47 +05:30
litao	c491f81a30	HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119 ). Contributed by tomscut. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:20:52 +05:30
LeonGao	643dfd60e4	HDFS-15842. HDFS mover to emit metrics. (#2738 )	2021-06-19 15:39:46 -07:00
hust_hhb	56d249759f	HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065 )	2021-06-17 11:08:29 +08:00
Wei-Chiu Chuang	a77bf7cf07	Make upstream aware of 3.3.1 release	2021-06-15 15:43:36 +00:00
Viraj Jasani	c1bf3cb0da	HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998 ) Contributed by Viraj Jasani	2021-06-14 14:23:50 -07:00
Siyao Meng	ebee2aed00	HDFS-16055. Quota is not preserved in snapshot INode (#3078 )	2021-06-14 10:48:22 -07:00
Ahmed Hussein	352949d070	HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097 )	2021-06-11 11:00:42 -05:00
litao	51ebf7b2a0	HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084 ) Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-06-11 17:30:55 +09:00
Viraj Jasani	4ef27a596f	HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087 )	2021-06-11 12:15:52 +09:00
Viraj Jasani	eefa664fea	HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073 )	2021-06-09 15:12:48 +09:00
Ahmed Hussein	a2a0283c7b	HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058 )	2021-06-08 11:09:31 -05:00
litao	c748fce17a	HDFS-16048. RBF: Print network topology on the router web (#3062 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-08 16:14:06 +09:00
huhaiyang	b38b00e528	HDFS-15998. Fix NullPointException In listOpenFiles (#3036 ) Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>	2021-06-01 00:26:47 -07:00
Ayush Saxena	b8ab19373d	Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 )" This reverts commit `041488e8f5`.	2021-06-01 08:38:13 +05:30
Ayush Saxena	bce14e746b	HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng.	2021-06-01 08:34:56 +05:30
Konstantin V Shvachko	8ce30f51f9	HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.	2021-05-27 18:47:20 -07:00
Konstantin V Shvachko	1abd03d68f	HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.	2021-05-26 14:59:50 -07:00
Hui Fei	98a74e2351	HDFS-16041. TestErasureCodingCLI fails (#3052 )	2021-05-26 20:12:17 +09:00
Viraj Jasani	986d0a4f1d	HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-26 17:14:31 +09:00
Narges Shadab	3fdeb7435a	HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027 )	2021-05-26 14:43:41 +09:00
Viraj Jasani	028ec4704b	HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-25 10:55:58 +09:00
huhaiyang	acd712ca92	HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747 )	2021-05-19 09:04:41 +08:00
Ayush Saxena	fcd4140e5f	HDFS-15988. Stabilise HDFS Pre-Commit. (#2860 ). Contributed by Ayush Saxena. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-05-16 00:45:11 +05:30
Viraj Jasani	b944084b32	HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-11 12:38:15 +09:00
Siyao Meng	8d5cc98b42	HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958 )	2021-05-10 12:41:07 -07:00
Ayush Saxena	359c0c1d1e	HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w.	2021-05-10 17:09:51 +05:30
Akira Ajisaka	9aa651f0b4	HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980 ) Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>	2021-05-08 00:24:06 +09:00
zhuqi	76c69c39d9	HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833 ) Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-05-05 21:02:13 +05:30
Viraj Jasani	041488e8f5	HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 ) Reviewed-by: Siyao Meng <siyao@apache.org>	2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang	f1e1809029	HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945 ) Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang	b2e54762a4	HDFS-15624. fix the function of setting quota by storage type (#2377 ) (#2955 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org> Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44	2021-04-28 23:54:39 -07:00
bshashikant	ef13f8ad6b	HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881 )	2021-04-27 09:15:06 +05:30
Stephen O'Donnell	605ed85c29	HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849 ). Contributed by Stephen O'Donnell	2021-04-26 11:00:23 +01:00
litao	b968fa0957	HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-26 18:38:43 +09:00
Takanobu Asanuma	01bad0e92a	HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel.	2021-04-26 13:29:28 +09:00
Brahma Reddy Battula	2621d3f15b	HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang	90c6caf650	Revert "HDFS-15624. fix the function of setting quota by storage type (#2377 )" This reverts commit `394b9f7a5c`. Ref: HDFS-15995. Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3) Will re-work this fix again later.	2021-04-26 11:27:15 +08:00
jianghuazhu	c486dc396c	HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913 ) Contributed by JiangHua Zhu.	2021-04-25 14:31:51 +08:00
Viraj Jasani	acb49f2fc3	HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-21 19:45:49 +09:00
litao	85a3532849	HDFS-15970. Print network topology on the web (#2896 )	2021-04-19 20:42:30 +09:00
litao	af0448d37b	HDFS-15975. Use LongAdder instead of AtomicLong (#2907 )	2021-04-17 16:49:09 +09:00
zhangshuyan0	14816be0b1	HDFS-15963. Unreleased volume references cause an infinite loop. (#2889 ) Contributed by Shuyan Zhang. Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-17 00:08:31 +08:00
Vivek Ratnavel Subramanian	c821008836	HDFS-15850. Superuser actions should be reported to external enforcers (#2784 )	2021-04-15 14:45:51 -07:00
Viraj Jasani	56bd968fb4	HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874 )	2021-04-09 11:10:18 +09:00
Stephen O'Donnell	4c567fcff7	HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838 )	2021-04-08 11:59:02 +01:00
He Xiaoqiao	459df41997	HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.	2021-04-06 21:22:48 +08:00
Viraj Jasani	3f2682b92b	HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 13:39:10 +09:00
Viraj Jasani	de05cefbae	HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844 ). Contributed by Viraj Jasani Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 00:19:21 +05:30
zhuqi	96e410a127	HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829 ). Contributed by Qi Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-04 03:11:02 +05:30
litao	3cb76447f5	HDFS-15946. Fix java doc in FSPermissionChecker (#2855 ). Contributed by tomscut. Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-02 10:37:04 -07:00
Brahma Reddy Battula	ed74479ea5	HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree.	2021-04-02 09:47:20 +05:30
litao	7c83f140dc	HDFS-15938. Fix java doc in FSEditLog (#2837 ). Contributed by tomscut. Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 19:28:17 -07:00
litao	4bd04126d6	HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-02 10:33:40 +09:00
Viraj Jasani	4f28738010	HDFS-15931 : Fix non-static inner classes for better memory management (#2830 ). Contributed by Viraj Jasani Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 16:34:31 -07:00
Brahma Reddy Battula	0665ce9930	HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.	2021-04-01 09:19:39 +05:30
He Xiaoqiao	5eca748970	HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.	2021-04-01 11:32:03 +08:00
Viraj Jasani	90fdd04f54	HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-29 23:22:54 +05:30
Ayush Saxena	cd2501e54b	HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.	2021-03-28 16:13:59 +05:30
litao	72037a63b1	HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748 ) Reviewed-by: Dinesh Chitlangia <dineshc@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2021-03-27 17:57:09 +09:00
He Xiaoqiao	ed5814deff	HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.	2021-03-25 12:24:14 +08:00
touchida	95e6892675	HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585 )	2021-03-24 16:56:09 +08:00
Viraj Jasani	4b4ccce02f	HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 22:47:45 -07:00
Ayush Saxena	569e407f64	HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.	2021-03-24 02:10:02 +05:30
Jack Jiang	9c2a712597	HDFS-15913. Remove useless NULL checks before instanceof (#2805 )	2021-03-23 08:51:35 -07:00
Narges Shadab	d77c7ab4e0	HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790 ). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 17:06:58 +05:30
litao	343ce8a8c9	HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788 )	2021-03-22 20:29:17 +09:00
He Xiaoqiao	03a3a04c06	HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-22 18:37:00 +08:00
Viraj Jasani	261191cbc0	HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-19 20:37:44 -07:00
Narges Shadab	f5f3fc6fe9	HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736 ). Contributed by Narges Shadab. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-19 16:36:48 +08:00
Viraj Jasani	9c43b60348	HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772 )	2021-03-17 15:12:36 +09:00
Takanobu Asanuma	9ba60c33d8	HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.	2021-03-17 14:14:26 +09:00
touchida	25390babc7	HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726 ) Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-03-17 12:00:05 +09:00
Hui Fei	0d42104119	HDFS-15898. Test case TestOfflineImageViewer fails (#2779 )	2021-03-17 09:11:30 +08:00
sunlisheng	7025f39944	HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.	2021-03-15 11:34:13 +08:00
He Xiaoqiao	970455c917	HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.	2021-03-14 17:48:18 +08:00
lamberken	6921ec8b40	HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751 ). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-13 14:22:11 +08:00
Xiaoyu Yao	7cf5969f0b	HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764 )	2021-03-12 11:27:53 -08:00
Takanobu Asanuma	ecd3335187	HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.	2021-03-12 15:42:15 +09:00
Hui Fei	6a55baeee4	HDFS-15875. Check whether file is being truncated before truncate (#2746 )	2021-03-10 14:11:29 +08:00
litao	6a922f9896	HDFS-15873. Add namenode address in logs for block report (#2743 )	2021-03-08 11:50:51 -05:00
tomscut	9cb51bf106	HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668 ) Contributed by tomscut.	2021-03-06 13:40:44 -08:00
litao	cdba06e380	HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739 ) Co-authored-by: tomscut <litao@bigo.sg>	2021-03-03 22:34:02 +09:00
zhuqi	1f1a1ef52d	HDFS-15856: Make write pipeline retry times configurable. (#2721 ). Contributed by Qi Zhu Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-02 13:16:11 +08:00
Konstantin V Shvachko	9501c698f4	HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.	2021-03-01 17:17:23 -08:00
litao	32353eb38a	HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718 ) Authored-by: tomscut <litao@bigo.sg>	2021-03-01 07:52:59 -08:00
S O'Donnell	c3b3b36dee	HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell	2021-03-01 11:36:41 +00:00
S O'Donnell	9c583f36d5	HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee	2021-02-24 10:29:36 +00:00
LeonGao	940c780feb	HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704 )	2021-02-23 22:29:48 -08:00
Ayush Saxena	647d23b611	HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.	2021-02-23 19:08:38 +05:30
Takanobu Asanuma	a57282540f	HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714 )	2021-02-23 15:12:06 +09:00
Wang Yuxuan	1e32ab7d2f	HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang	f8059d0bdf	Revert "first commit (#2559 )" This reverts commit `1b94569d89`.	2021-02-22 18:13:06 +08:00
Wang Yuxuan	1b94569d89	first commit (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 02:10:39 -08:00
jianghuazhu	98ffd65d8b	HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>	2021-02-21 23:22:32 -08:00
Ayush Saxena	ff84a57483	HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.	2021-02-20 19:48:00 +05:30
Takanobu Asanuma	25af8901a9	HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.	2021-02-19 14:44:58 +09:00
Hui Fei	2970bd93f3	HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694 )	2021-02-19 09:07:22 +08:00
Ayush Saxena	e391844e8e	HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.	2021-02-18 01:39:04 +05:30
Akira Ajisaka	0ed9e36282	HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696 )	2021-02-17 01:59:24 -05:00
Zehao Chen	07a4220cd2	HDFS-15821. Add metrics for in-service datanodes (#2690 ). Contributed by Zehao Chen.	2021-02-15 11:14:32 -06:00
litao	c3134ab3a9	HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676 )	2021-02-14 00:12:24 -05:00
Steve Loughran	798df6d699	HADOOP-13327 Output Stream Specification. (#2587 ) This defines what output streams and especially those which implement Syncable are meant to do, and documents where implementations (HDFS; S3) don't. With tests. The file:// FileSystem now supports Syncable if an application calls FileSystem.setWriteChecksum(false) before creating a file -checksumming and Syncable.hsync() are incompatible. Contributed by Steve Loughran.	2021-02-10 10:28:59 +00:00
LeonGao	9434c1eccc	HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679 )	2021-02-09 10:49:28 +00:00
bshashikant	2df2dfb9ed	HDFS-15817. Rename snapshots while marking them deleted. (#2677 )	2021-02-09 10:27:51 +05:30
LeonGao	0e2b3086e3	HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625 )	2021-02-08 11:01:21 -08:00
bshashikant	19ae0faacc	HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682 )	2021-02-06 08:26:12 -08:00
Jim Brennan	c4918fb298	[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan	2021-02-05 21:20:30 +00:00
fermi	7ef2875c89	HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang	2021-02-05 11:01:49 +08:00
maobaolong	b99dba38b9	HDFS-15819. Fix import style #2681	2021-02-05 10:11:27 +08:00
Hui Fei	15a1f7adfc	HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang	2021-02-04 09:19:51 +08:00
Kihwal Lee	182623d2bc	HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.	2021-02-03 08:55:50 -06:00
Hui Fei	66ecee333e	HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang	2021-02-03 20:00:45 +08:00
YaYun-Wang	394b9f7a5c	HDFS-15624. fix the function of setting quota by storage type (#2377 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>	2021-02-02 22:44:34 -08:00
crossfire	18978f2e20	HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657 )	2021-02-02 09:02:09 +00:00
He Xiaoqiao	1b893e10ae	HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-02 13:50:54 +08:00
belugabehr	21a3fc3d2d	HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633 )	2021-02-01 09:40:01 -08:00
He Xiaoqiao	b54134661b	HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-01 23:28:29 +08:00
Narges Shadab	115623a6ee	HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652 )	2021-02-01 01:56:59 -08:00
Siyao Meng	1a205cc3ad	HADOOP-17424. Replace HTrace with No-Op tracer (#2645 )	2021-02-01 13:42:44 +09:00
Jim Brennan	f565476939	[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.	2021-01-28 18:17:15 +00:00
Jim Brennan	6184ab07f6	[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.	2021-01-28 16:47:52 +00:00
sunlisheng	f8769e0f4b	HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.	2021-01-28 10:10:39 +08:00
Akira Ajisaka	45b3a84709	HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626 )	2021-01-21 10:07:46 +09:00
He Xiaoqiao	97f843de3a	HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.	2021-01-16 14:06:56 +08:00
LeonGao	7743d40ac5	HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583 ). Contributed by Leon Gao.	2021-01-15 16:28:11 -08:00
Masatake Iwasaki	630f8ddd2c	HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-14 16:19:35 +09:00
touchida	8ec824f2ba	HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598 ) Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-13 11:23:07 +09:00
Masatake Iwasaki	32abc0af49	HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507 )	2021-01-11 11:59:32 -08:00
He Xiaoqiao	ec22850dbe	Make upstream aware of 3.2.2 release.	2021-01-09 18:06:08 +08:00
Ahmed Hussein	77435a025e	HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601 ). Contributed by Ahmed Hussein and Daryn Sharp.	2021-01-08 13:10:09 -06:00
lfengnan	87bd4d2aca	HDFS-15754. Add DataNode packet metrics (#2578 ) Contributed by Fengnan Li.	2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang	66ee0a6df0	HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453 )	2021-01-04 09:43:58 -08:00
Masatake Iwasaki	2825d060cf	HDFS-15648. TestFileChecksum should be parameterized. (#2501 )	2021-01-04 14:00:57 +09:00
Ayush Saxena	1cd96e8dd8	HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.	2021-01-01 11:37:52 +05:30
maobaolong	3b77cf4446	HDFS-15749. Make size of editPendingQ can be configurable (#2572 )	2020-12-28 14:18:18 +08:00
hemanthboyina	16a20503ca	HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.	2020-12-22 17:27:31 +05:30
langlaile1221	4ae561bcb4	HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566 )	2020-12-22 09:25:11 +08:00
Ayush Saxena	ba496f61f8	HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.	2020-12-22 02:16:00 +05:30
Ayush Saxena	2aea43bf4f	HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.	2020-12-20 23:06:19 +05:30
Ayush Saxena	3ecd3628d4	HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.	2020-12-20 15:18:25 +05:30
Ayush Saxena	70789fc6b9	HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.	2020-12-20 15:12:32 +05:30
Ahmed Hussein	c2672bb234	HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-17 11:33:29 -06:00
Ayush Saxena	0da00fc743	HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-12-14 20:42:23 -08:00
liuyanpunk	3363cc380b	HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534 )	2020-12-15 10:06:11 +05:30
S O'Donnell	9ed737001c	HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell	2020-12-11 20:53:10 +00:00
Ahmed Hussein	be35fa186c	HDFS-15717. Improve fsck logging. (#2529 ) Contributed by Kihwal Lee and Ahmed Hussein	2020-12-11 11:02:42 -06:00
Neil	9bd3c9bc50	HDFS-15720 namenode audit async logger should add some log4j config (#2532 )	2020-12-10 13:47:00 -08:00
Ahmed Hussein	01383a2172	HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528 )	2020-12-08 13:25:24 -06:00
crossfire	40f7543a6d	HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518 )	2020-12-07 15:49:45 -08:00
Ahmed Hussein	32099e36dd	HDFS-15707. NNTop counts don't add up as expected. (#2516 )	2020-12-07 14:22:06 -06:00
Ayush Saxena	ad40715690	HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.	2020-12-07 15:00:19 +05:30
Hui Fei	e2c1268ebd	HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.	2020-12-04 09:20:09 +08:00
Wei-Chiu Chuang	42a29199c0	HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.	2020-12-02 21:32:02 -08:00
LeonGao	6ff2409b31	HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483 ). Contributed by Leon Gao.	2020-12-02 15:53:09 -08:00
Ahmed Hussein	60201cbf69	HDFS-15703. Don't generate edits for set operations that are no-op (#2508 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 15:38:20 -06:00
Ahmed Hussein	2b5b556dd7	HDFS-15695. NN should not let the balancer run in safemode (#2489 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 13:59:00 -06:00
Ahmed Hussein	918ba9e82f	HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487 ) Contributed by Kuhu Shukla and Ahmed Hussein	2020-12-01 15:02:53 -06:00
Hui Fei	68442b48c8	HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495 )	2020-11-28 12:19:57 +08:00
Siyao Meng	235947e282	HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472 )	2020-11-25 11:01:04 -08:00
He Xiaoqiao	641d8856d2	HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh	d73029463c	HDFS-15690. Add lz4-java as test dependency (#2481 )	2020-11-21 17:49:56 -08:00
Ahmed Hussein	747883af4d	HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443 ). Contributed by Ahmed Hussein.	2020-11-20 18:03:45 +05:30
Masatake Iwasaki	425996eb4a	HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467 )	2020-11-18 16:11:09 +09:00
Takanobu Asanuma	f56cd88d7d	HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.	2020-11-13 11:02:26 +09:00
Issac Buenrostro	f5e6be337b	HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255 ) Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>	2020-11-12 14:57:26 -08:00
jianghuazhu	e6d2dccbef	HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138 )	2020-11-10 16:34:17 -08:00
Yizhong Zhang	82de07c794	HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.	2020-11-10 09:11:31 -08:00
maobaolong	95c96605b3	HDFS-15667. Audit log record the unexpected allowed result when delete (#2437 )	2020-11-10 13:01:10 +08:00
LeonGao	9a9ab5b48e	HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288 ). Contributed by Leon Gao.	2020-11-09 15:06:16 -08:00
He Xiaoqiao	3067a25fa1	HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Yiqun Lin <yqlin@apache.org>	2020-11-04 13:53:46 +08:00
Ayush Saxena	7ac7840fc3	HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424 ). Contributed by Ayush Saxena.	2020-11-04 09:28:44 +05:30
Konstantin V Shvachko	d07dc7afb4	HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.	2020-11-03 12:02:40 -08:00
Ahmed Hussein	324879127a	HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419 )	2020-10-28 16:24:34 -07:00
Ahmed Hussein	d0c786db4d	HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406 )	2020-10-27 18:56:40 -07:00
Ahmed Hussein	98097b8f19	HDFS-15457. TestFsDatasetImpl fails intermittently (#2407 )	2020-10-27 18:52:56 -07:00
Hui Fei	8a6d5b9151	HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416 )	2020-10-28 09:13:25 +08:00
Ahmed Hussein	ae74407ac4	HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 20:45:00 +09:00
Akira Ajisaka	3848b44b37	HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309 )	2020-10-27 19:18:08 +09:00
Ahmed Hussein	afaab3d332	HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 16:56:30 +09:00
Hui Fei	12c908c827	HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.	2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang	74634eb002	HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.	2020-10-23 12:05:53 -07:00
Inigo Goiri	e7aa4da0a2	HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.	2020-10-23 09:47:10 -07:00
Kihwal Lee	da1b6e3cc2	HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.	2020-10-22 16:59:49 -05:00
Kihwal Lee	88a9f42f32	HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.	2020-10-21 00:59:35 -05:00
Ayush Saxena	2e8cafac3b	HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.	2020-10-19 10:48:47 +05:30
Ayush Saxena	173310e2f5	HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.	2020-10-19 10:34:17 +05:30
Ayush Saxena	1e3a6efcef	HADOOP-17288. Use shaded guava from thirdparty. (#2342 ). Contributed by Ayush Saxena.	2020-10-17 12:01:18 +05:30
Kihwal Lee	740a2c4635	HDFS-15627. Log delete audits before collecting blocks.	2020-10-16 11:13:41 -05:00
Siyao Meng	a308a1ec22	HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370 )	2020-10-13 10:59:42 -07:00
Gabor Bota	59f01a548e	HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same Contributed by Gabor Bota.	2020-10-13 16:17:44 +01:00
Mukund Thakur	0507c4160f	HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur. Change-Id: I6cec41c20467920cf21f169653553535414b2680	2020-10-13 13:31:00 +01:00
Steve Loughran	d606ddc12c	Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 )" (reverting to change the JIRA number) This reverts commit `3e1b1dfd77`. Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd	2020-10-13 13:30:38 +01:00
Mukund Thakur	3e1b1dfd77	HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur.	2020-10-13 13:27:55 +01:00
Konstantin V Shvachko	b3786d6c3c	HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.	2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G	b76b36ebbc	HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378 ). Contributed by Uma Maheswara Rao G.	2020-10-12 14:22:44 -07:00
LeonGao91	de055ec8aa	HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299 )	2020-10-10 22:08:36 +05:30
Karthik Palanisamy	735e85a1d2	HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365 )	2020-10-08 12:51:04 +05:30
Karthik Palanisamy	df4006eb81	HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366 )	2020-10-07 22:09:04 +05:30
bshashikant	16aea11c94	HDFS-15611. Add list Snapshot command in WebHDFS. (#2355 )	2020-10-07 10:34:32 +05:30
Fei Hui	6ece6407e1	HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356 ). Contributed by Hui Fei.	2020-10-05 22:11:24 +05:30
Siyao Meng	43b0c0b054	HDFS-15607. Create trash dir when allowing snapshottable dir (#2352 )	2020-10-05 05:02:00 -07:00
Fei Hui	984875c624	HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345 )	2020-09-28 09:54:01 -07:00
Takanobu Asanuma	d89af79507	HDFS-15577. Refactor TestTracing. (#2302 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2020-09-28 18:27:41 +09:00
Vinayakumar B	82b86e3754	HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng	2020-09-27 19:27:13 +05:30
Ye Ni	00c4de63cf	HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332 )	2020-09-25 09:47:54 -07:00
Aryan Gupta	486ddb73f9	HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2020-09-24 21:15:55 +09:00
huangtianhua	ff59fbb8b0	HDFS-15025. Applying NVDIMM storage media to HDFS (#2189 ) * HDFS-15025. Applying NVDIMM storage media to HDFS Co-authored-by: YaYun-Wang <yywangyayun@163.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>	2020-09-24 14:27:04 +05:30
bshashikant	368f2f637e	HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326 )	2020-09-24 14:00:41 +05:30
Ye Ni	ae089f2db7	HDFS-15557. Log the reason why a storage log file can't be deleted (#2274 )	2020-09-22 13:23:20 -07:00
Hui Fei	474fa80bfb	HADOOP-17277. Correct spelling errors for separator (#2322 ) Contributed by Hui Fei.	2020-09-22 17:10:33 +01:00
Ayush Saxena	2377df38ad	HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.	2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G	7bba4c609c	HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312 ). Contributed by Uma Maheswara Rao G.	2020-09-18 15:18:10 +05:30
S O'Donnell	20a0e6278d	HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell	2020-09-17 14:11:42 +01:00
bshashikant	425f48799c	HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296 )	2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G	e4cb0d3514	HDFS-15578: Fix the rename issues with fallback fs enabled (#2305 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-16 22:43:00 -07:00
Takanobu Asanuma	5c5b2ed7c7	HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.	2020-09-16 01:36:46 +09:00
hemanthboyina	f4ed9f3f91	HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.	2020-09-15 10:10:21 +05:30
He Xiaoqiao	924959088e	HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.	2020-09-15 12:05:04 +08:00
Siyao Meng	6d3312de47	HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258 )	2020-09-14 13:31:34 -07:00
Uma Maheswara Rao G	d2779de3f5	HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298 ). Contributed by Uma Maheswara Rao G.	2020-09-12 17:06:39 -07:00
Mingliang Liu	f59f7f2175	HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell	2020-09-12 01:41:38 -07:00
hemanthboyina	2a6fdcedeb	HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.	2020-09-11 18:21:25 +05:30
1996fanrui	56ebabd426	HADOOP-17222. Create socket address leveraging URI cache (#2241 ) Contributed by fanrui. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-09-10 22:30:52 -07:00
zz	12a316cdf9	HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185 ). Contributed by Zhenzhao Wang. Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>	2020-09-10 21:20:32 -07:00
Siyao Meng	43c52d6495	HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir (#2295 ) Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc	2020-09-10 15:55:23 +05:30
He Xiaoqiao	1d6d0d8207	HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun. Co-authored-by: Chen Zhang <chzhang1987@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2020-09-09 20:09:13 +08:00
Uma Maheswara Rao G	ac7d4623ae	HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275 ) Contributed by Uma Maheswara Rao G.	2020-09-07 11:36:13 -07:00
Uma Maheswara Rao G	b3660d0147	HDFS-15529: getChildFilesystems should include fallback fs as well (#2234 ). Contributed by Uma Maheswara Rao G.	2020-09-03 11:06:20 -07:00
He Xiaoqiao	60de592a88	HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.	2020-08-31 14:49:12 +08:00
bshashikant	2d03209a06	HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion (#2251 )	2020-08-31 11:29:48 +05:30
Steve Loughran	c392d9022a	HDFS-15471. TestHDFSContractMultipartUploader failing (#2252 ) Contributed by Steve Loughran (Was: broken by Steve Loughran)	2020-08-28 15:39:09 +01:00
Vivek Ratnavel Subramanian	06793da100	HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory (#2230 )	2020-08-27 11:36:30 -07:00
Tsz-Wo Nicholas Sze	41182a9b6d	HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. (#2233 )	2020-08-27 14:54:52 +05:30
Stephen O'Donnell	2ffe00fc46	HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-08-26 23:05:32 -07:00
S O'Donnell	6a49bf9bff	HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.	2020-08-25 15:18:36 +01:00
He Xiaoqiao	a7830423c5	HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.	2020-08-24 21:16:00 +08:00
hemanthboyina	83fd15b412	HDFS-14504. Rename with Snapshots does not honor quota limit.	2020-08-23 16:10:47 +05:30
Chen Liang	f734455e5d	HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.	2020-08-20 15:25:10 -07:00
Uma Maheswara Rao G	dd013f2fdf	HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229 ). Contributed by Uma Maheswara Rao G.	2020-08-19 09:30:41 -07:00
Siyao Meng	b65e43fe38	HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS (#2220 )	2020-08-18 19:28:19 +09:00
bshashikant	092bfe7c8e	HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. (#2172 ) * HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. * Addressed review comments. * Rebased to latest trunk and added snapshotTrashRoot config check. * Addressed review comments. * Removede empty line added in SnapshotManager.Java. * Addressed whitespace issues.	2020-08-17 12:56:13 +05:30
Ayush Saxena	e3d1966f58	HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.	2020-08-15 15:07:58 +05:30
Tsz-Wo Nicholas Sze	b93dd7c281	HDFS-15519. Check inaccessible INodes in FsImageValidation. (#2224 )	2020-08-14 10:10:01 -07:00
bshashikant	15a76e8d65	HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. (#2219 )	2020-08-14 20:46:48 +05:30
Vivek Ratnavel Subramanian	cb50e3fcf7	HDFS-15496. Add UI for deleted snapshots (#2212 )	2020-08-13 22:36:15 +05:30
Aryan Gupta	e592ec5f8b	HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. (#2217 ) * HDFS-15518. Fixed String operationName = ListSnapshot. * HDFS-15518. Changed ListSnapshot to operationName. Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>	2020-08-13 00:09:38 +05:30
S O'Donnell	10716040a8	HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang	2020-08-12 09:02:47 +01:00
Tsz-Wo Nicholas Sze	11cec9ab94	HDFS-15523. Fix findbugs warnings from HDFS-15520. (#2218 )	2020-08-11 21:41:28 -07:00
Siyao Meng	3fd3aeb621	HDFS-15492. Make trash root inside each snapshottable directory (#2176 )	2020-08-11 08:52:16 -07:00
Uma Maheswara Rao G	8955a6ceb7	HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205 ) * HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false * Used LambdaTestUtils#intercept in test	2020-08-11 00:01:58 -07:00
Tsz-Wo Nicholas Sze	592127bdc2	HDFS-15520 Use visitor pattern to visit namespace tree (#2203 )	2020-08-11 12:30:30 +05:30
Akira Ajisaka	7938ebfb9d	HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.	2020-08-11 11:14:58 +09:00
Ayush Saxena	c2a17659d1	HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.	2020-08-08 14:33:53 +05:30
Ayush Saxena	975b6024dd	HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.	2020-08-07 22:19:17 +05:30
bshashikant	e072d33327	HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable (#2175 )	2020-08-04 14:10:29 +05:30
hemanthboyina	82f3ffcd64	HDFS-15503. File and directory permissions are not able to be modified from WebUI.	2020-08-03 23:21:01 +05:30
Akira Ajisaka	c40cbc57fa	HADOOP-17091. [JDK11] Fix Javadoc errors (#2098 )	2020-08-03 10:46:51 +09:00
bshashikant	d8a2df25ad	HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181 )	2020-08-02 21:59:56 +05:30
hemanthboyina	528a799a78	HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.	2020-08-02 12:02:37 +05:30
Tsz-Wo Nicholas Sze	05b3337a46	HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots (#2165 )	2020-07-30 23:06:51 +05:30
bshashikant	68287371cc	HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166 )	2020-07-29 21:33:25 +05:30
touchida	026dce5334	HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. (#2135 )	2020-07-27 09:55:11 -07:00
bshashikant	2d12496643	HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (#2163 )	2020-07-22 16:16:27 -07:00
Uma Maheswara Rao G	ac9a07b51a	HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160 ). Contributed by Uma Maheswara Rao G.	2020-07-21 23:29:10 -07:00
Masatake Iwasaki	1b29c9bfee	HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155 )	2020-07-22 13:40:20 +09:00
Inigo Goiri	8b7695bb26	HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.	2020-07-21 09:00:07 -07:00
Tsz-Wo Nicholas Sze	d57462f2da	HDFS-15479. Ordered snapshot deletion: make it a configurable feature (#2156 )	2020-07-21 11:36:24 +05:30
Shashikant Banerjee	d9441f95c3	HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.	2020-07-21 11:13:05 +05:30
Chen Liang	3833c616e0	HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.	2020-07-20 12:49:58 -07:00
Ayush Saxena	6cbd8854ee	HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.	2020-07-20 21:49:05 +05:30
Tsz-Wo Nicholas Sze	2cec50cf16	HDFS-15463. Add a tool to validate FsImage. (#2140 )	2020-07-20 11:44:30 +05:30
Ayush Saxena	6bcb24d269	HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.	2020-07-18 14:33:43 +05:30
Ahmed Hussein	1f71c4ae71	HADOOP-17099. Replace Guava Predicate with Java8+ Predicate Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 11:39:06 -05:00
Ahmed Hussein	98fcffe93f	HADOOP-17101. Replace Guava Function with Java8+ Function Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 09:53:18 -05:00
Shashikant Banerjee	85d4718ed7	HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.	2020-07-15 10:32:25 +05:30
zhaoyim	bdce75d737	HDFS-15371. Nonstandard characters exist in NameNode.java (#2032 ) Contributed by zhaoyim	2020-07-14 14:42:12 +01:00
He Xiaoqiao	b97fea65e7	HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.	2020-07-13 14:12:48 +08:00
Uma Maheswara Rao G	3e70006639	HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132 ). Contributed by Uma Maheswara Rao G.	2020-07-11 23:50:04 -07:00
Xiaoyu Yao	f91a8ad88b	HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085 )	2020-07-09 11:33:37 -07:00
Ye Ni	f77bbc2123	HDFS-15312. Apply umask when creating directory by WebHDFS (#2096 )	2020-07-06 19:01:46 -07:00
Uma Maheswara Rao G	dc0626b5f2	HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.	2020-07-06 18:50:03 -07:00
Shanyu Zhao	834372f404	HDFS-15451. Do not discard non-initial block report for provided storage. (#2119 ). Contributed by Shanyu Zhao. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-07-06 23:43:34 +08:00
Uma Maheswara Rao G	55a2ae80dc	HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.	2020-07-04 13:45:49 -07:00
Ayush Saxena	f86f15cf20	HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.	2020-07-04 13:00:43 +05:30
Uma Maheswara Rao G	1f2a80b5e5	HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-07-04 00:12:10 -07:00
He Xiaoqiao	9ac498e300	HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.	2020-07-01 12:38:07 +08:00
Stephen O'Donnell	2a67e2b1a0	HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-30 07:10:17 -07:00
Akira Ajisaka	c71ce7ac33	HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.	2020-06-28 16:02:47 +09:00
Ayush Saxena	8db38c98a6	HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.	2020-06-27 19:13:01 +05:30
Virajith Jalaparti	bed0a3a374	HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100 ) * HDFS-15436. Default mount table name used by ViewFileSystem should be configurable * Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests * Address Uma's comments on PR#2100 * Sort lists in test to match without concern to order * Address comments, fix checkstyle and fix failing tests * Fix checkstyle	2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G	d5e1bb6155	HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-06-26 01:29:38 -07:00
Uma Maheswara Rao G	7c02d1889b	HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.	2020-06-23 01:42:25 -07:00
Ayush Saxena	17ffcab5f6	HDFS-14546. Document block placement policies. Contributed by Amithsha.	2020-06-21 17:27:28 +05:30
Uma Maheswara Rao G	b27810aa60	HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.	2020-06-20 00:32:02 -07:00
Stephen O'Donnell	d50e93ce7b	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang	edf716a5c3	Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." This reverts commit `730a39d138`.	2020-06-18 06:45:15 -07:00
S O'Donnell	123777823e	HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina	2020-06-18 12:26:22 +01:00
Ayush Saxena	c8ed33cd2a	HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.	2020-06-17 13:55:40 +05:30
Stephen O'Donnell	730a39d138	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-16 15:59:13 -07:00
Takanobu Asanuma	f41a144077	HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.	2020-06-15 09:15:53 +09:00
Inigo Goiri	719b53a79d	HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.	2020-06-13 09:35:05 -07:00
Uma Maheswara Rao G	785b1def95	HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.	2020-06-12 14:32:19 -07:00
Vinayakumar B	e154084770	HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026 )	2020-06-12 23:16:33 +05:30
Ayush Saxena	b735a77717	HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.	2020-06-10 12:06:16 +05:30
Ayush Saxena	8525874561	HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. *Added missed test file.	2020-06-09 18:53:33 +05:30
Uma Maheswara Rao G	3ca15292c5	HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.	2020-06-06 08:11:57 -07:00
Ayush Saxena	cc671b16f7	HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena	2020-06-05 22:19:38 -07:00
Uma Maheswara Rao G	76fa0222f0	HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.	2020-06-05 10:58:21 -07:00
Ayush Saxena	2326123705	HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.	2020-06-05 19:15:05 +05:30
Toshihiro Suzuki	545a0a147c	HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052 ) Contributed by Toshihiro Suzuki.	2020-06-05 11:11:49 +01:00
Ayush Saxena	e8cb2ae409	HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.	2020-06-03 12:47:15 +05:30
Ayush Saxena	f6453244ab	HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan.	2020-06-03 12:16:36 +05:30
Uma Maheswara Rao G	ed83c865dd	HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.	2020-06-02 11:09:26 -07:00
Ayush Saxena	ae13a5ccbe	HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.	2020-05-31 17:10:59 +05:30
Wanqiang Ji	d9838f2d42	HADOOP-17055. Remove residual code of Ozone (#2039 )	2020-05-29 16:49:18 +09:00
Ayush Saxena	a838d871a7	HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He.	2020-05-28 20:52:58 +05:30
Ayush Saxena	9b38be43c6	HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.	2020-05-28 13:00:04 +05:30
Inigo Goiri	2148a8fe64	HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina.	2020-05-27 11:06:13 -07:00
Ayush Saxena	6c9f75cf16	HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena.	2020-05-26 16:26:57 +05:30
Ayush Saxena	f43a152b97	HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun.	2020-05-24 18:03:41 +05:30
Ayush Saxena	f4901d0778	HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun.	2020-05-24 17:59:16 +05:30
Inigo Goiri	6e04b00df1	HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.	2020-05-23 09:58:19 -07:00
Ayush Saxena	e0ae232f66	HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.	2020-05-23 18:38:32 +05:30
Ayush Saxena	f99fcb26ab	HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.	2020-05-23 18:29:31 +05:30
Takanobu Asanuma	4d22d1c58f	HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.	2020-05-23 17:28:38 +09:00
Uma Maheswara Rao G	4734c77b4b	HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.	2020-05-21 21:34:58 -07:00
leosunli	be374faf42	HDFS-13639. SlotReleaser is not fast enough (#1885 )	2020-05-21 13:21:17 -07:00
Abhishek Das	ce4ec74453	HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.	2020-05-18 22:27:12 -07:00
Chen Liang	7bb902bc0d	HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang	2020-05-18 10:58:52 -07:00
Ayush Saxena	c84e6beada	HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena.	2020-05-18 22:24:34 +05:30
Wei-Chiu Chuang	2abcf7762a	HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 )	2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang	4525292d41	Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 )" This reverts commit `86e6aa8eec`.	2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang	50caba1a92	HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun.	2020-05-18 08:40:38 -07:00
He Xiaoqiao	a3f44dacc1	HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-05-18 07:09:43 -07:00
pustota2009	86e6aa8eec	HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 ) Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.	2020-05-18 07:04:04 -07:00
Ayush Saxena	6e416a83d1	HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.	2020-05-17 03:06:27 +05:30
Ayush Saxena	178336f8a8	HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He.	2020-05-16 16:35:06 +05:30
Uma Maheswara Rao G	ac4a2e11d9	HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.	2020-05-14 17:29:35 -07:00
Surendra Singh Lilhore	1958cb7c2b	YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng	2020-05-14 15:09:34 +05:30
Ayush Saxena	0918433b4d	YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng.	2020-05-14 00:36:20 +05:30
Xiaoyu Yao	3cacf1ce56	HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. (#2004 )	2020-05-13 11:47:19 -07:00
Surendra Singh Lilhore	743c2e9071	HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina	2020-05-13 15:01:07 +05:30
Elixir Kook	a3f945fb84	HADOOP-17035. fixed typos (timeout, interruped) (#2007 ) Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>	2020-05-12 10:50:04 -05:00
S O'Donnell	29dddb8a14	HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.	2020-05-12 15:07:51 +01:00
Takanobu Asanuma	928b81a533	HDFS-15350. Set dfs.client.failover.random.order to true as default. (#2008 )	2020-05-12 09:04:03 -05:00
Ayush Saxena	8dad38c0be	HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.	2020-05-12 18:34:26 +05:30
Ayush Saxena	0fe49036e5	HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.	2020-05-12 13:11:31 +05:30
Ayush Saxena	4c53fb9ce1	HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.	2020-05-11 16:48:34 +05:30
Ayush Saxena	cb64e993c2	HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi.	2020-05-10 00:11:46 +05:30
Ayush Saxena	d59de27c01	HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.	2020-05-08 00:53:49 +05:30
Inigo Goiri	130f89e068	HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.	2020-05-06 13:18:32 -07:00
Ayush Saxena	7fddf4855e	HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.	2020-05-06 16:55:04 +05:30
Wei-Chiu Chuang	adecdb8b53	HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998 ) Reviewed-by: Arpit Agarwal <arp@apache.org>	2020-05-05 16:04:54 -07:00
Uma Maheswara Rao G	9c8236d04d	HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.	2020-05-04 17:55:40 -07:00
Konstantin V Shvachko	ebb878bab9	HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.	2020-05-04 10:29:50 -07:00
Ayush Saxena	8dace8ff3a	HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi.	2020-05-03 23:59:17 +05:30
Ayush Saxena	d6fc482a54	Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.	2020-05-03 19:09:14 +05:30
Mingliang Liu	44de193bec	HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li	2020-05-02 20:56:29 -07:00
Akira Ajisaka	f40dacdc2e	HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992 ) Signed-off-by: Mingliang Liu <liuml07@apache.org>	2020-05-02 13:02:27 -07:00
Shashikant Banerjee	82343790ee	HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.	2020-05-01 12:16:02 +05:30
Mingliang Liu	263c76b678	HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2020-04-30 14:15:28 -07:00
S O'Donnell	9ca6298a9a	HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.	2020-04-29 15:56:06 +01:00
Akira Ajisaka	5e0eda5d5f	HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.	2020-04-28 06:17:14 +09:00
Toshihiro Suzuki	62c26b91fd	HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 (#1979 )	2020-04-27 17:20:47 +09:00
Mingliang Liu	74d4a18735	HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena	2020-04-25 15:43:02 -07:00
Dhiraj	453c9c9a67	HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964 ) Contributed by Dhiraj Hegde. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Inigo Goiri <inigoiri@apache.org>	2020-04-25 13:04:32 -07:00
Wei-Chiu Chuang	453771f170	HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.	2020-04-24 11:53:24 -07:00
Mingliang Liu	5958af4dbc	HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina	2020-04-22 13:31:02 -07:00
Toshihiro Suzuki	1824aee9da	HDFS-15217 Add more information to longest write/read lock held log	2020-04-18 13:52:07 -07:00
Ayush Saxena	37d6582223	HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.	2020-04-17 13:32:02 +05:30
Stephen O'Donnell	3481895f8a	HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-04-16 20:18:25 -07:00
Ayush Saxena	cc5c1da7c1	HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang.	2020-04-15 23:01:58 +05:30
Wei-Chiu Chuang	061afcdf30	HDFS-15269. NameNode should check the authorization API version only … (#1945 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2020-04-09 09:19:35 -07:00
Ayush Saxena	1189af4746	HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena.	2020-04-08 15:55:14 +05:30
Toshihiro Suzuki	c12ddbd1de	HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922 )	2020-04-07 14:38:09 +09:00
Inigo Goiri	0b855b9f35	HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun.	2020-04-06 13:05:33 -07:00
He Xiaoqiao	d3b5951572	HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2020-04-01 16:36:46 -07:00
Ayush Saxena	960c9ebaea	HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang.	2020-03-30 02:40:25 +05:30
Brahma Reddy Battula	8914cf9167	Preparing for 3.4.0 development	2020-03-29 23:24:25 +05:30
Inigo Goiri	f7a94ec0a4	HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina.	2020-03-29 08:17:02 -07:00
Ayush Saxena	696a663cd0	HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest.	2020-03-29 00:01:22 +05:30
Chen Liang	f531a4a487	HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.	2020-03-27 12:55:38 -07:00
Wei-Chiu Chuang	0fa7bf47df	HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909 )	2020-03-25 16:03:26 -07:00
Inigo Goiri	cdcb77a2c5	HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He.	2020-03-25 11:30:54 -07:00
Arpit Agarwal	a700803a18	HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348	2020-03-25 10:28:30 -07:00
Toshihiro Suzuki	d353b30baf	HDFS-15215. The Timestamp for longest write/read lock held log is wrong	2020-03-24 14:50:15 -07:00
Ayush Saxena	ea87d60493	HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun.	2020-03-24 22:56:09 +05:30
Karthik Palanisamy	5250cd6db3	HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870 )	2020-03-24 15:15:46 +05:30
He Xiaoqiao	af64ce2f4a	HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-03-23 12:43:51 -07:00
Ayush Saxena	62c8015683	HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena.	2020-03-22 11:44:18 +05:30
Takanobu Asanuma	b74e47e31a	HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.	2020-03-22 01:44:55 +09:00
Ayush Saxena	f9bb2a8cc5	HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.	2020-03-19 22:26:34 +05:30
Ayush Saxena	bb41ddaf1e	HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.	2020-03-19 21:23:13 +05:30
Ayush Saxena	654db35fa2	HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.	2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang	096533c2dc	HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. Contributed by Wei-Chiu Chuang. Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d	2020-03-18 13:27:13 +00:00
Ayush Saxena	8a7c54995a	HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina.	2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang	7b637a3341	HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.	2020-03-13 19:01:23 -07:00
He Xiaoqiao	e9955bb8ff	HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Brahma Reddy Battula <brahma@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang	4b95c242ec	HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829 ) Reviewed-by: Xiaoyu Yao <xyao@apache.org>	2020-03-13 11:29:44 -07:00
Ayush Saxena	fb1d7635ae	HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang.	2020-03-13 21:10:03 +05:30
S O'Donnell	25497799cb	HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.	2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang	20903f72b4	HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun.	2020-03-12 19:23:12 -07:00
Inigo Goiri	0695f7a538	HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang.	2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore	f736408a83	HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.	2020-03-12 18:59:17 +05:30
Steve Loughran	c734d69a55	HADOOP-16898. Batch listing of multiple directories via an (unstable) interface Contributed by Steve Loughran. This moves the new API of HDFS-13616 into a interface which is implemented by HDFS RPC filesystem client (not WebHDFS or any other connector) This new interface, BatchListingOperations, is in hadoop-common, so applications do not need to be compiled with HDFS on the classpath. They must cast the FS into the interface. instanceof can probe the client for having the new interface -the patch also adds a new path capability to probe for this. The FileSystem implementation is cut; tests updated as appropriate. All new interfaces/classes/constants are marked as @unstable. Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681	2020-03-09 14:51:16 +00:00
Inigo Goiri	4062217189	HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena.	2020-03-06 10:14:53 -08:00
Inigo Goiri	3afd4cbe89	HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina.	2020-03-04 10:13:23 -08:00
Inigo Goiri	1d4d0fcbe1	HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.	2020-03-04 10:02:54 -08:00
Akshay Nehe	bbd704bb82	HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871 )	2020-03-04 11:07:31 +09:00
Ayush Saxena	1a636da041	HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun.	2020-02-29 09:55:29 +05:30
Inigo Goiri	97b797c314	HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.	2020-02-28 18:47:22 -08:00
Surendra Singh Lilhore	fccfb02672	HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.	2020-02-28 18:33:06 +05:30
Ayush Saxena	429da635ec	HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong.	2020-02-28 00:31:55 +05:30
Ayush Saxena	cd2c6b1aac	HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest.	2020-02-27 22:19:35 +05:30
Surendra Singh Lilhore	2059f255d3	HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena.	2020-02-27 14:08:42 +05:30
Ayush Saxena	033a3d7ff9	HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.	2020-02-27 04:02:26 +05:30
Ayush Saxena	209630472a	HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun.	2020-02-27 02:22:24 +05:30
Kihwal Lee	352a4ec16d	HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein.	2020-02-26 09:33:29 -06:00
Stephen O'Donnell	900430b990	HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang	1c5d2f1fdc	HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.	2020-02-24 13:39:16 -08:00
Ayush Saxena	7f8685f476	HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena.	2020-02-24 20:38:04 +05:30
Ayush Saxena	93b8f453b9	HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li.	2020-02-24 20:17:01 +05:30
Ayush Saxena	b5698e0c33	HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun.	2020-02-24 00:07:18 +05:30
Ayush Saxena	9eb7a8bdf8	HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi.	2020-02-23 14:25:39 +05:30
Ayush Saxena	ba9025c7cd	HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun.	2020-02-22 14:27:26 +05:30

... 8 9 10 11 12 ...

9024 Commits