hadoop

Author	SHA1	Message	Date
Ahmed Hussein	bddc9bf63c	HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375 )	2021-09-08 21:21:16 -07:00
Symious	c0890e6d04	HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379 )	2021-09-09 09:57:27 +08:00
litao	5a306955a4	HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378 )	2021-09-08 12:38:55 +08:00
jianghuazhu	4e4d4fcc7c	HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368 )	2021-09-06 18:06:12 +08:00
Renukaprasad C	dd6b2cebed	HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244 ). Contributed by Renukaprasad C.	2021-09-05 17:07:59 +08:00
litao	364a1fbfb3	HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354 ). Contributed by tomscut.	2021-09-05 14:19:01 +08:00
Viraj Jasani	cc6b545365	HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-04 16:20:08 +05:30
Weison Wei	a610f6d9c3	HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367 ). Contributed by Weison Wei. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-03 12:05:53 +05:30
langlaile1221	24af52efab	HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894 )	2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G	164608b546	HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353 ) Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2021-08-31 12:25:03 +08:00
Viraj Jasani	aa9cdf2af6	HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-26 00:37:38 -07:00
LeonGao	b53cae0ffb	HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284 ) * Add DNS resolution for QJM * Add log * Resolve comments * checkstyle * typo	2021-08-25 17:40:12 -07:00
Viraj Jasani	1b9927afe1	HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-08-25 17:37:25 +09:00
Neil	9084c728eb	HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315 )	2021-08-24 11:15:47 +08:00
Gautham B A	b6d1971820	HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311 )	2021-08-20 10:04:55 -07:00
litao	07627ef19e	HDFS-16177. Bug fix for Util#receiveFile (#3310 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-08-19 12:30:38 +08:00
litao	0e922aab41	HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291 )	2021-08-19 09:51:00 +08:00
Gautham B A	409878ef80	HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303 )	2021-08-17 09:49:52 -07:00
jianghuazhu	53a2c65694	HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-17 16:47:10 +08:00
Viraj Jasani	6342d5e523	HDFS-16171. De-flake testDecommissionStatus (#3280 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-16 14:54:25 +09:00
Viraj Jasani	919a99bbb4	HDFS-16163. Avoid locking entire blockPinningFailures map (#3296 )	2021-08-16 13:40:05 +08:00
Gautham B A	77383a4d55	HDFS-15976. Make mkdtemp cross platform (#2908 )	2021-08-10 09:23:28 -07:00
Akira Ajisaka	4972e7a246	Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk.	2021-08-07 11:27:25 +09:00
wangzhaohui	e85c44657c	HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270 ) Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-06 18:23:00 +08:00
wangzhaohui	a73b64f86b	HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269 ). Contributed by wangzhaohui.	2021-08-06 09:27:05 +01:00
zhangshuyan0	10a2526b0b	HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247 ) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-04 00:22:21 +08:00
Viraj Jasani	ccfa072dc7	HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang	6d77f3b6cd	HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-30 08:29:56 -07:00
jianghuazhu	6f730fd25c	HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: cxorm <lianp964@gmail.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-07-28 21:57:28 -07:00
He Xiaoqiao	683feaa1d4	HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>	2021-07-29 11:19:17 +08:00
S O'Donnell	4eae284827	HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell	2021-07-28 14:49:23 +01:00
zhihaixu2012	ae20516ebc	HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175 ) Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a Co-authored-by: Zhihai Xu <zxu@apache.org>	2021-07-27 19:18:44 -07:00
JiaguodongF	aecfcf165f	HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185 ) * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-27 22:25:21 +05:30
Viraj Jasani	b038042ece	HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228 ). Contributed by Viraj Jasani. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-27 14:57:12 +08:00
Akira Ajisaka	d710ec8d85	HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-07-26 16:46:07 +09:00
litao	5d765497c5	HDFS-16131. Show storage type for failed volumes on namenode web (#3211 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-26 12:45:01 +08:00
Akira Ajisaka	2f2f822488	HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227 ) Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin" This reverts commit `b6d839a60c`. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml	2021-07-26 01:53:39 +09:00
Eric Yin	de41ce8a16	HDFS-16087. Fix stuck issue in rbfbalance tool (#3141 ). Contributed by Eric Yin.	2021-07-21 00:01:55 +08:00
He Xiaoqiao	6ed7670a93	HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.	2021-07-17 21:57:00 +08:00
Kihwal Lee	47002719f2	HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee.	2021-07-16 14:22:39 -05:00
crossfire	4bb25c810b	HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414 )	2021-07-15 19:58:12 -07:00
Abhishek Das	1dd03cc4b5	HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260 )	2021-07-13 18:11:50 -07:00
LeonGao	ea90c5117d	HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639 ) * Rebase trunk * Fix to use FQDN and update config name * Fix javac * Style and trigger build * Trigger Build after force push * Trigger Build * Fix config names	2021-07-13 14:18:59 -07:00
Artem Smotrakov	ba325a8ada	HADOOP-17793. Better token validation (#3189 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-10 13:42:31 +09:00
S O'Donnell	b6c06c4b76	HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma	2021-07-09 10:03:57 +01:00
bshashikant	6450c1ea9a	HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188 )	2021-07-09 10:26:42 +05:30
Viraj Jasani	618c9218ee	HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171 ) Reviewed-by: Steve Loughran <stevel@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-07-08 16:03:40 +09:00
litao	b4c2647d0d	HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140 )	2021-07-08 14:10:45 +08:00
JiaguodongF	9381071758	HDFS-16114. the balancer parameters print error (#3179 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-07 14:13:05 +05:30
Ayush Saxena	c85a18327d	HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w.	2021-07-06 23:26:52 +05:30
litao	a447cfd78b	HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-07-05 19:31:22 -07:00
Viraj Jasani	da575a53cf	HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169 )	2021-07-05 09:23:24 +08:00
litao	b87bac13e4	HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-05 08:14:37 +09:00
litao	390f8603d3	HDFS-16106. Fix flaky unit test TestDFSShell (#3168 )	2021-07-02 15:31:25 +08:00
zhangshuyan0	9be17339eb	HDFS-16099. Make bpServiceToActive to be volatile. (#3159 ) Co-authored-by: zhangshuyan <zqingchai@gmail.com>	2021-07-01 23:52:21 +08:00
litao	52b9319e57	HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-01 18:44:10 +05:30
litao	56c7ada7a5	HDFS-16086. Add volume information to datanode log for tracing (#3136 )	2021-06-30 22:06:56 -07:00
Viraj Jasani	d2e7003400	HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148 ) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: stack Chuang <stack@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-30 12:55:07 -07:00
Xiangyi Zhu	4cac6ec405	HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156 ) Co-authored-by: zhuxiangyi <43412979@qq.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-06-30 00:12:08 -07:00
litao	95454d821c	HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146 )	2021-06-29 03:15:12 -07:00
Viraj Jasani	93a1685073	HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150 )	2021-06-29 02:31:02 -07:00
He Xiaoqiao	748570b73c	Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." This reverts commit `0d07837712`.	2021-06-25 22:38:34 +08:00
He Xiaoqiao	0d07837712	HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun.	2021-06-25 01:09:42 +08:00
litao	f78b6d47f9	HDFS-16085. Move the getPermissionChecker out of the read lock (#3134 ). Contributed by tomscut. Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-24 15:59:42 +05:30
litao	fdef2b4cca	HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117 )	2021-06-24 11:27:40 +09:00
Gautham B A	ef5dbc7283	HADOOP-17766. CI for Debian 10 (#3129 )	2021-06-23 10:02:33 -07:00
Viraj Jasani	d9fbb3c508	HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127 )	2021-06-22 20:21:34 -07:00
He Xiaoqiao	10b79a26fe	HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. Co-authored-by: Tao Yang <taoyang1@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-06-22 00:32:55 +08:00
Viraj Jasani	c488abbc79	HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-06-21 10:25:12 +09:00
litao	bdc9c8809e	HDFS-16079. Improve the block state change log (#3120 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 17:23:11 +05:30
Viraj Jasani	d9eccb1eb3	HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 17:19:05 +05:30
Ahmed Hussein	c255feb5d7	HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095 ). Contributed by Ahmed Hussein. Reviewed-by: Jim Brennan <jbrennan@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:22:47 +05:30
litao	c491f81a30	HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119 ). Contributed by tomscut. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:20:52 +05:30
LeonGao	643dfd60e4	HDFS-15842. HDFS mover to emit metrics. (#2738 )	2021-06-19 15:39:46 -07:00
Wei-Chiu Chuang	1ad674e548	HDFS-16074. Remove an expensive debug string concatenation (#3107 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: litao <tomleescut@gmail.com>	2021-06-16 20:37:56 -07:00
hust_hhb	56d249759f	HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065 )	2021-06-17 11:08:29 +08:00
Ayush Saxena	2b304ad645	HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w.	2021-06-16 15:53:12 +05:30
Wei-Chiu Chuang	a77bf7cf07	Make upstream aware of 3.3.1 release	2021-06-15 15:43:36 +00:00
Takanobu Asanuma	839fcf7682	HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104 )	2021-06-15 13:20:09 +05:30
Viraj Jasani	c1bf3cb0da	HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998 ) Contributed by Viraj Jasani	2021-06-14 14:23:50 -07:00
Siyao Meng	ebee2aed00	HDFS-16055. Quota is not preserved in snapshot INode (#3078 )	2021-06-14 10:48:22 -07:00
Ahmed Hussein	352949d070	HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097 )	2021-06-11 11:00:42 -05:00
litao	51ebf7b2a0	HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084 ) Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-06-11 17:30:55 +09:00
Viraj Jasani	4ef27a596f	HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087 )	2021-06-11 12:15:52 +09:00
Viraj Jasani	eefa664fea	HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073 )	2021-06-09 15:12:48 +09:00
Ayush Saxena	a1a3184171	HDFS-15916. Addendum. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#3056 )	2021-06-08 19:34:45 -07:00
Ahmed Hussein	a2a0283c7b	HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058 )	2021-06-08 11:09:31 -05:00
litao	c748fce17a	HDFS-16048. RBF: Print network topology on the router web (#3062 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-08 16:14:06 +09:00
huhaiyang	b38b00e528	HDFS-15998. Fix NullPointException In listOpenFiles (#3036 ) Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>	2021-06-01 00:26:47 -07:00
Ayush Saxena	b8ab19373d	Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 )" This reverts commit `041488e8f5`.	2021-06-01 08:38:13 +05:30
Ayush Saxena	bce14e746b	HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng.	2021-06-01 08:34:56 +05:30
Jinglun	e90c41af34	HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-30 19:28:40 +08:00
Jinglun	2b11e160e2	Revert "HDFS-15973." because the user.name and user.email message is not correct. This reverts commit `b7f69467c1`.	2021-05-30 19:27:10 +08:00
李经纶	b7f69467c1	HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-30 18:50:03 +08:00
Konstantin V Shvachko	8ce30f51f9	HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.	2021-05-27 18:47:20 -07:00
Konstantin V Shvachko	1abd03d68f	HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.	2021-05-26 14:59:50 -07:00
Gautham B A	bcaeb1ac8c	Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898 )"" (#3044 )	2021-05-26 14:12:53 -07:00
Hui Fei	98a74e2351	HDFS-16041. TestErasureCodingCLI fails (#3052 )	2021-05-26 20:12:17 +09:00
Viraj Jasani	986d0a4f1d	HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-26 17:14:31 +09:00
Narges Shadab	3fdeb7435a	HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027 )	2021-05-26 14:43:41 +09:00
zhuxiangyi	e18d806212	HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009 )	2021-05-26 13:41:49 +08:00
Viraj Jasani	028ec4704b	HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-25 10:55:58 +09:00
lfengnan	43bf009112	HDFS-15757 RBF: Improving Router Connection Management (#2651 )	2021-05-19 10:53:42 -07:00
huhaiyang	acd712ca92	HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747 )	2021-05-19 09:04:41 +08:00
Ayush Saxena	fcd4140e5f	HDFS-15988. Stabilise HDFS Pre-Commit. (#2860 ). Contributed by Ayush Saxena. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-05-16 00:45:11 +05:30
Viraj Jasani	b944084b32	HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-11 12:38:15 +09:00
Siyao Meng	8d5cc98b42	HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958 )	2021-05-10 12:41:07 -07:00
Ayush Saxena	359c0c1d1e	HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w.	2021-05-10 17:09:51 +05:30
Jinglun	1b6994287a	HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819 ). Contributed by zhuobin zheng. Reviewed-by: Jinglun <jinglun@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-08 19:50:17 +08:00
Akira Ajisaka	9aa651f0b4	HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980 ) Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>	2021-05-08 00:24:06 +09:00
zhuqi	76c69c39d9	HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833 ) Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-05-05 21:02:13 +05:30
Viraj Jasani	041488e8f5	HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 ) Reviewed-by: Siyao Meng <siyao@apache.org>	2021-05-04 01:31:23 -07:00
Akira Ajisaka	7c561e2a02	HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970 )	2021-05-04 00:25:29 +09:00
lfengnan	6e525ab81c	HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang	f1e1809029	HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945 ) Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-02 11:10:32 +09:00
Karthik Palanisamy	bd3da73a0f	HDFS-15865. Interrupt DataStreamer thread if no ack (#2728 )	2021-05-01 23:35:31 +05:30
lfengnan	552e9dcc6c	HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954 ). Contributed by Fengnan Li. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang	b2e54762a4	HDFS-15624. fix the function of setting quota by storage type (#2377 ) (#2955 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org> Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44	2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang	f41a368c14	HADOOP-11245. Update NFS gateway to use Netty4 (#2832 ) Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>	2021-04-28 23:43:11 -07:00
bshashikant	ef13f8ad6b	HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881 )	2021-04-27 09:15:06 +05:30
Stephen O'Donnell	605ed85c29	HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849 ). Contributed by Stephen O'Donnell	2021-04-26 11:00:23 +01:00
litao	b968fa0957	HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-26 18:38:43 +09:00
Takanobu Asanuma	01bad0e92a	HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel.	2021-04-26 13:29:28 +09:00
Brahma Reddy Battula	2621d3f15b	HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang	90c6caf650	Revert "HDFS-15624. fix the function of setting quota by storage type (#2377 )" This reverts commit `394b9f7a5c`. Ref: HDFS-15995. Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3) Will re-work this fix again later.	2021-04-26 11:27:15 +08:00
jianghuazhu	c486dc396c	HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913 ) Contributed by JiangHua Zhu.	2021-04-25 14:31:51 +08:00
zhuxiangyi	dff95c5eca	HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915 ) Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-23 10:19:11 +09:00
Viraj Jasani	acb49f2fc3	HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-21 19:45:49 +09:00
litao	85a3532849	HDFS-15970. Print network topology on the web (#2896 )	2021-04-19 20:42:30 +09:00
litao	af0448d37b	HDFS-15975. Use LongAdder instead of AtomicLong (#2907 )	2021-04-17 16:49:09 +09:00
Eric Badger	aed13f0f42	Revert "HDFS-15971. Make mkstemp cross platform (#2898 )" This reverts commit `b088f46b68`.	2021-04-16 17:15:33 +00:00
zhangshuyan0	14816be0b1	HDFS-15963. Unreleased volume references cause an infinite loop. (#2889 ) Contributed by Shuyan Zhang. Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-17 00:08:31 +08:00
Akira Ajisaka	f0241ec216	HDFS-15977. Call explicit_bzero only if it is available. (#2914 ) Reviewed-by: Masatake Iwasaki <iwasakims@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-04-16 13:26:45 +09:00
lfengnan	17be99f9f9	HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903 )	2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian	c821008836	HDFS-15850. Superuser actions should be reported to external enforcers (#2784 )	2021-04-15 14:45:51 -07:00
Masatake Iwasaki	2717203f85	HADOOP-17569. Building native code fails on Fedora 33. (#2886 )	2021-04-15 21:33:18 +09:00
Ayush Saxena	7d6f0ca0bd	Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754 ). Contributed by tomscut." This reverts commit `e565b05c80`. The removed method needs to be used in HDFS-15423.	2021-04-13 23:28:42 +05:30
Gautham B A	b088f46b68	HDFS-15971. Make mkstemp cross platform (#2898 )	2021-04-13 08:54:45 -07:00
Inigo Goiri	82462739f8	Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605 )" (#2900 ) This reverts commit `cb3ed32fe0`.	2021-04-12 20:44:31 -07:00
lfengnan	cb3ed32fe0	HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605 )	2021-04-12 12:42:33 -07:00
Gautham B A	3148791da4	HDFS-15962. Make strcasecmp cross platform (#2883 )	2021-04-09 09:01:33 -07:00
Viraj Jasani	56bd968fb4	HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874 )	2021-04-09 11:10:18 +09:00
Gautham B A	bf66116407	HDFS-15955. Make explicit_bzero cross platform (#2875 )	2021-04-08 09:44:47 -07:00
Ayush Saxena	c6539e3289	HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863 ). Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-04-08 20:49:08 +05:30
Stephen O'Donnell	4c567fcff7	HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838 )	2021-04-08 11:59:02 +01:00
Gautham B A	42ddb5c6fe	HDFS-15909. Make fnmatch cross platform (#2792 )	2021-04-06 09:23:21 -07:00
He Xiaoqiao	459df41997	HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.	2021-04-06 21:22:48 +08:00
Gautham B A	36014b8282	HDFS-15949. Fix integer overflow (#2857 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-04-06 18:22:54 +09:00
Viraj Jasani	3f2682b92b	HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 13:39:10 +09:00
Viraj Jasani	de05cefbae	HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844 ). Contributed by Viraj Jasani Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 00:19:21 +05:30
Gautham B A	6e1df1c048	HDFS-15947. Replace deprecated protobuf APIs (#2856 )	2021-04-05 09:52:48 -07:00
litao	9e2d5d6d05	HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-05 17:47:10 +09:00
Gautham B A	ca5e13fd74	HDFS-15950. Remove unused hdfs.proto import (#2858 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-04-05 11:42:05 +09:00
zhuqi	96e410a127	HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829 ). Contributed by Qi Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-04 03:11:02 +05:30
litao	3cb76447f5	HDFS-15946. Fix java doc in FSPermissionChecker (#2855 ). Contributed by tomscut. Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-02 10:37:04 -07:00
Gautham B A	70536ba1f9	HDFS-15944. Prevent truncation by snprintf (#2853 )	2021-04-02 09:37:33 -07:00
Brahma Reddy Battula	ed74479ea5	HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree.	2021-04-02 09:47:20 +05:30
litao	7c83f140dc	HDFS-15938. Fix java doc in FSEditLog (#2837 ). Contributed by tomscut. Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 19:28:17 -07:00
litao	4bd04126d6	HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-02 10:33:40 +09:00
Viraj Jasani	4f28738010	HDFS-15931 : Fix non-static inner classes for better memory management (#2830 ). Contributed by Viraj Jasani Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 16:34:31 -07:00
Brahma Reddy Battula	0665ce9930	HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.	2021-04-01 09:19:39 +05:30
He Xiaoqiao	5eca748970	HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.	2021-04-01 11:32:03 +08:00
Borislav Iordanov	2c482fbacf	HADOOP-16524. Automatic keystore reloading for HttpServer2 Reapply of issue reverted first because it caused yarn failures and then again because the commit message was incorrectly formatted (and yet again because of commit message format). Signed-off-by: stack <stack@apache.org>	2021-03-31 10:46:35 -07:00
stack	22961a615d	Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" This reverts commit `a2975d2153`.	2021-03-31 10:43:09 -07:00
stack	a2975d2153	HADOOP-16524. Automatic keystore reloading for HttpServer2 Reapply of issue reverted first because it caused yarn failures and then again because the commit message was incorrectly formatted.	2021-03-31 10:40:20 -07:00
stack	5183aaeda2	Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693 )" Revert to fix the summary message. This reverts commit `9509bebf7f`.	2021-03-31 10:39:55 -07:00
Borislav Iordanov	9509bebf7f	Hadoop 16524 - resubmission following some unit test fixes (#2693 ) Signed-off-by: stack <stack@apache.org>	2021-03-31 10:07:42 -07:00
He Xiaoqiao	da4ceba4aa	HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C.	2021-03-31 13:00:13 +08:00
Gautham B A	7dfff496fa	HDFS-15929. Replace RAND_pseudo_bytes in util.cc (#2826 )	2021-03-30 13:36:02 -07:00
Gautham B A	8668abf87e	HDFS-15927. Catch polymorphic type by reference (#2824 )	2021-03-30 13:35:45 -07:00
Gautham B A	50b1e1c0c4	HDFS-15928. Replace RAND_pseudo_bytes in rpc_engine.cc (#2825 )	2021-03-30 10:37:10 -07:00
Viraj Jasani	90fdd04f54	HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-29 23:22:54 +05:30
Gautham B A	6a3e135316	HDFS-15922. Use memcpy for copying non-null terminated string. (#2827 )	2021-03-29 10:38:59 -07:00
Harunobu Daikoku	ea6595d3b6	HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787 ) Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>	2021-03-29 11:43:29 +09:00
Ayush Saxena	cd2501e54b	HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.	2021-03-28 16:13:59 +05:30
litao	72037a63b1	HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748 ) Reviewed-by: Dinesh Chitlangia <dineshc@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2021-03-27 17:57:09 +09:00
Gautham B A	1fed18bb2d	HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c (#2818 )	2021-03-26 09:26:35 -07:00
Akira Ajisaka	af1f9f43ea	HADOOP-17133. Implement HttpServer2 metrics (#2145 )	2021-03-25 12:09:43 -07:00
He Xiaoqiao	ed5814deff	HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.	2021-03-25 12:24:14 +08:00
Gautham B A	654555783d	HDFS-15918. Replace deprecated RAND_pseudo_bytes (#2811 )	2021-03-24 10:52:33 -07:00
touchida	95e6892675	HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585 )	2021-03-24 16:56:09 +08:00
Takanobu Asanuma	b4d97a8dc7	HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel.	2021-03-24 17:51:35 +09:00
Viraj Jasani	4b4ccce02f	HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 22:47:45 -07:00
Ayush Saxena	569e407f64	HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.	2021-03-24 02:10:02 +05:30
Gautham B A	714427c414	HDFS-15910. Improve security with explicit_bzero (#2793 )	2021-03-23 09:58:19 -07:00
Jack Jiang	9c2a712597	HDFS-15913. Remove useless NULL checks before instanceof (#2805 )	2021-03-23 08:51:35 -07:00
Narges Shadab	d77c7ab4e0	HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790 ). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 17:06:58 +05:30
Gautham B A	61bfa42389	HDFS-15903. Refactor X-Platform lib (#2783 )	2021-03-22 08:40:54 -07:00
litao	343ce8a8c9	HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788 )	2021-03-22 20:29:17 +09:00
He Xiaoqiao	03a3a04c06	HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-22 18:37:00 +08:00
Viraj Jasani	261191cbc0	HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-19 20:37:44 -07:00
Narges Shadab	f5f3fc6fe9	HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736 ). Contributed by Narges Shadab. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-19 16:36:48 +08:00
Viraj Jasani	9c43b60348	HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772 )	2021-03-17 15:12:36 +09:00
Takanobu Asanuma	9ba60c33d8	HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.	2021-03-17 14:14:26 +09:00
touchida	25390babc7	HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726 ) Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-03-17 12:00:05 +09:00
Hui Fei	0d42104119	HDFS-15898. Test case TestOfflineImageViewer fails (#2779 )	2021-03-17 09:11:30 +08:00
Gautham B A	47620f8821	HDFS-15843. Make write cross-platform (#2710 )	2021-03-16 09:06:51 -07:00
sunlisheng	7025f39944	HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.	2021-03-15 11:34:13 +08:00
He Xiaoqiao	970455c917	HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.	2021-03-14 17:48:18 +08:00
litao	e565b05c80	HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754 ). Contributed by tomscut.	2021-03-14 04:07:31 +05:30
lamberken	6921ec8b40	HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751 ). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-13 14:22:11 +08:00
Xiaoyu Yao	7cf5969f0b	HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764 )	2021-03-12 11:27:53 -08:00
Takanobu Asanuma	ecd3335187	HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.	2021-03-12 15:42:15 +09:00
Akira Ajisaka	23b343aed1	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753 ) Removed findbugs from the hadoop build images and added spotbugs instead. Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>	2021-03-11 10:56:07 +09:00
Hui Fei	6a55baeee4	HDFS-15875. Check whether file is being truncated before truncate (#2746 )	2021-03-10 14:11:29 +08:00
litao	6a922f9896	HDFS-15873. Add namenode address in logs for block report (#2743 )	2021-03-08 11:50:51 -05:00
He Xiaoqiao	4a0b7f7ebe	HDFS-15384. Document getLocatedBlocks(String src, long start) of DFSClient only return partial blocks. Contributed by Yang Yun.	2021-03-08 12:18:46 +08:00
tomscut	9cb51bf106	HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668 ) Contributed by tomscut.	2021-03-06 13:40:44 -08:00
litao	cdba06e380	HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739 ) Co-authored-by: tomscut <litao@bigo.sg>	2021-03-03 22:34:02 +09:00
JJiaguodong	8af56de1fa	HADOOP-17560. Fix some spelling errors (#2730 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-03-03 11:41:05 +09:00
zhuqi	1f1a1ef52d	HDFS-15856: Make write pipeline retry times configurable. (#2721 ). Contributed by Qi Zhu Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-02 13:16:11 +08:00
Konstantin V Shvachko	9501c698f4	HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.	2021-03-01 17:17:23 -08:00
litao	32353eb38a	HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718 ) Authored-by: tomscut <litao@bigo.sg>	2021-03-01 07:52:59 -08:00
S O'Donnell	c3b3b36dee	HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell	2021-03-01 11:36:41 +00:00
S O'Donnell	9c583f36d5	HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee	2021-02-24 10:29:36 +00:00
LeonGao	940c780feb	HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704 )	2021-02-23 22:29:48 -08:00
Ayush Saxena	647d23b611	HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.	2021-02-23 19:08:38 +05:30
Takanobu Asanuma	a57282540f	HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714 )	2021-02-23 15:12:06 +09:00
Wang Yuxuan	1e32ab7d2f	HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang	f8059d0bdf	Revert "first commit (#2559 )" This reverts commit `1b94569d89`.	2021-02-22 18:13:06 +08:00
Wang Yuxuan	1b94569d89	first commit (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 02:10:39 -08:00
jianghuazhu	98ffd65d8b	HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>	2021-02-21 23:22:32 -08:00
Ayush Saxena	ff84a57483	HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.	2021-02-20 19:48:00 +05:30
Ayush Saxena	2ce5752fa8	HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun.	2021-02-20 19:34:29 +05:30
Takanobu Asanuma	25af8901a9	HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.	2021-02-19 14:44:58 +09:00
Akira Ajisaka	9a298d180d	Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 )" This reverts commit `4cf3531583`.	2021-02-19 11:09:10 +09:00
Hui Fei	2970bd93f3	HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694 )	2021-02-19 09:07:22 +08:00
Ayush Saxena	e391844e8e	HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.	2021-02-18 01:39:04 +05:30
Akira Ajisaka	0ed9e36282	HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696 )	2021-02-17 01:59:24 -05:00
Akira Ajisaka	4cf3531583	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 ) Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images, and added spotbugs in the images instead. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>	2021-02-17 10:38:20 +09:00
Akira Ajisaka	2b3c5b1733	HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-02-16 14:50:50 +09:00
Zehao Chen	07a4220cd2	HDFS-15821. Add metrics for in-service datanodes (#2690 ). Contributed by Zehao Chen.	2021-02-15 11:14:32 -06:00
litao	c3134ab3a9	HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676 )	2021-02-14 00:12:24 -05:00
Steve Loughran	798df6d699	HADOOP-13327 Output Stream Specification. (#2587 ) This defines what output streams and especially those which implement Syncable are meant to do, and documents where implementations (HDFS; S3) don't. With tests. The file:// FileSystem now supports Syncable if an application calls FileSystem.setWriteChecksum(false) before creating a file -checksumming and Syncable.hsync() are incompatible. Contributed by Steve Loughran.	2021-02-10 10:28:59 +00:00
LeonGao	9434c1eccc	HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679 )	2021-02-09 10:49:28 +00:00
bshashikant	2df2dfb9ed	HDFS-15817. Rename snapshots while marking them deleted. (#2677 )	2021-02-09 10:27:51 +05:30
LeonGao	0e2b3086e3	HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625 )	2021-02-08 11:01:21 -08:00
bshashikant	19ae0faacc	HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682 )	2021-02-06 08:26:12 -08:00
Jim Brennan	c4918fb298	[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan	2021-02-05 21:20:30 +00:00
fermi	7ef2875c89	HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang	2021-02-05 11:01:49 +08:00
maobaolong	b99dba38b9	HDFS-15819. Fix import style #2681	2021-02-05 10:11:27 +08:00
Hui Fei	15a1f7adfc	HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang	2021-02-04 09:19:51 +08:00
Kihwal Lee	182623d2bc	HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.	2021-02-03 08:55:50 -06:00
Hui Fei	66ecee333e	HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang	2021-02-03 20:00:45 +08:00
YaYun-Wang	394b9f7a5c	HDFS-15624. fix the function of setting quota by storage type (#2377 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>	2021-02-02 22:44:34 -08:00
crossfire	18978f2e20	HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657 )	2021-02-02 09:02:09 +00:00
He Xiaoqiao	1b893e10ae	HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-02 13:50:54 +08:00
belugabehr	21a3fc3d2d	HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633 )	2021-02-01 09:40:01 -08:00
He Xiaoqiao	b54134661b	HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-01 23:28:29 +08:00
Narges Shadab	115623a6ee	HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652 )	2021-02-01 01:56:59 -08:00
Siyao Meng	1a205cc3ad	HADOOP-17424. Replace HTrace with No-Op tracer (#2645 )	2021-02-01 13:42:44 +09:00
Gautham B A	ad483fd66e	HDFS-15740. Add x-platform utilities (#2567 )	2021-01-29 10:11:38 -08:00
Jim Brennan	f565476939	[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.	2021-01-28 18:17:15 +00:00
Jim Brennan	6184ab07f6	[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.	2021-01-28 16:47:52 +00:00
sunlisheng	f8769e0f4b	HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.	2021-01-28 10:10:39 +08:00
Ayush Saxena	e40f99f6d5	HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun.	2021-01-26 14:25:27 +05:30
Akira Ajisaka	45b3a84709	HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626 )	2021-01-21 10:07:46 +09:00
He Xiaoqiao	97f843de3a	HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.	2021-01-16 14:06:56 +08:00
LeonGao	7743d40ac5	HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583 ). Contributed by Leon Gao.	2021-01-15 16:28:11 -08:00
Masatake Iwasaki	630f8ddd2c	HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-14 16:19:35 +09:00
touchida	8ec824f2ba	HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598 ) Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-13 11:23:07 +09:00
Yiqun Lin	0d7ac54510	HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li.	2021-01-12 23:38:55 +08:00
Masatake Iwasaki	32abc0af49	HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507 )	2021-01-11 11:59:32 -08:00
He Xiaoqiao	ec22850dbe	Make upstream aware of 3.2.2 release.	2021-01-09 18:06:08 +08:00
Ahmed Hussein	77435a025e	HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601 ). Contributed by Ahmed Hussein and Daryn Sharp.	2021-01-08 13:10:09 -06:00
Ayush Saxena	2ba7ec2b48	HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun.	2021-01-08 16:27:54 +05:30
lfengnan	87bd4d2aca	HDFS-15754. Add DataNode packet metrics (#2578 ) Contributed by Fengnan Li.	2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang	2b4febcf57	HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533 )	2021-01-04 20:54:44 -08:00
Ayush Saxena	77299ae992	HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun.	2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang	66ee0a6df0	HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453 )	2021-01-04 09:43:58 -08:00
Masatake Iwasaki	2825d060cf	HDFS-15648. TestFileChecksum should be parameterized. (#2501 )	2021-01-04 14:00:57 +09:00
Ayush Saxena	1cd96e8dd8	HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.	2021-01-01 11:37:52 +05:30
maobaolong	3b77cf4446	HDFS-15749. Make size of editPendingQ can be configurable (#2572 )	2020-12-28 14:18:18 +08:00
hemanthboyina	16a20503ca	HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.	2020-12-22 17:27:31 +05:30
langlaile1221	4ae561bcb4	HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566 )	2020-12-22 09:25:11 +08:00
Ayush Saxena	ba496f61f8	HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.	2020-12-22 02:16:00 +05:30
Ayush Saxena	5ab1db9e21	HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut.	2020-12-21 09:58:30 +05:30
Ayush Saxena	2aea43bf4f	HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.	2020-12-20 23:06:19 +05:30
Ayush Saxena	3ecd3628d4	HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.	2020-12-20 15:18:25 +05:30
Ayush Saxena	70789fc6b9	HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.	2020-12-20 15:12:32 +05:30
Ahmed Hussein	c2672bb234	HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-17 11:33:29 -06:00
Ayush Saxena	0da00fc743	HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-12-14 20:42:23 -08:00
liuyanpunk	3363cc380b	HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534 )	2020-12-15 10:06:11 +05:30
S O'Donnell	9ed737001c	HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell	2020-12-11 20:53:10 +00:00
Ahmed Hussein	be35fa186c	HDFS-15717. Improve fsck logging. (#2529 ) Contributed by Kihwal Lee and Ahmed Hussein	2020-12-11 11:02:42 -06:00
Neil	9bd3c9bc50	HDFS-15720 namenode audit async logger should add some log4j config (#2532 )	2020-12-10 13:47:00 -08:00
Ahmed Hussein	3ec01b1bb3	HDFS-15711. Add Metrics to HttpFS Server. (#2521 ) Contributed by Ahmed Hussein and Kihwal Lee	2020-12-10 13:09:53 -06:00
Ahmed Hussein	01383a2172	HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528 )	2020-12-08 13:25:24 -06:00
Gautham B A	4ffec79b7c	HDFS-15712. Upgrade googletest to 1.10.0 (#2523 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-12-08 17:08:14 +09:00
crossfire	40f7543a6d	HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518 )	2020-12-07 15:49:45 -08:00
Ahmed Hussein	32099e36dd	HDFS-15707. NNTop counts don't add up as expected. (#2516 )	2020-12-07 14:22:06 -06:00
Yiqun Lin	da1ea2530f	HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.	2020-12-07 18:52:12 +08:00
Ayush Saxena	ad40715690	HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.	2020-12-07 15:00:19 +05:30
Ayush Saxena	7dda804a1a	HDFS-14090. RBF: Improved isolation for downstream name nodes. {Static}. Contributed by Fengnan Li and CR Hota.	2020-12-04 22:56:37 +05:30
Hui Fei	e2c1268ebd	HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.	2020-12-04 09:20:09 +08:00
Ahmed Hussein	07655a7040	HDFS-15706. HttpFS: Log more information on request failures. (#2515 )	2020-12-03 15:14:40 -08:00
Wei-Chiu Chuang	42a29199c0	HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.	2020-12-02 21:32:02 -08:00
LeonGao	6ff2409b31	HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483 ). Contributed by Leon Gao.	2020-12-02 15:53:09 -08:00
Ahmed Hussein	60201cbf69	HDFS-15703. Don't generate edits for set operations that are no-op (#2508 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 15:38:20 -06:00
Ahmed Hussein	2b5b556dd7	HDFS-15695. NN should not let the balancer run in safemode (#2489 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 13:59:00 -06:00
Ahmed Hussein	918ba9e82f	HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487 ) Contributed by Kuhu Shukla and Ahmed Hussein	2020-12-01 15:02:53 -06:00
Masatake Iwasaki	6a1d7d9ed2	HDFS-15677. TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk. (#2503 )	2020-12-01 08:06:52 +09:00
Hui Fei	68442b48c8	HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495 )	2020-11-28 12:19:57 +08:00
Akira Ajisaka	2ce2198287	HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2020-11-26 11:32:10 +09:00
Akira Ajisaka	65002c9486	Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488 )" because JIRA ID is missing. This reverts commit `ce5b3d7975`.	2020-11-26 11:30:53 +09:00
Akira Ajisaka	ce5b3d7975	[JDK 11] Fix error in mvn package -Pdocs (#2488 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2020-11-26 11:27:30 +09:00
Siyao Meng	235947e282	HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472 )	2020-11-25 11:01:04 -08:00
He Xiaoqiao	641d8856d2	HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh	d73029463c	HDFS-15690. Add lz4-java as test dependency (#2481 )	2020-11-21 17:49:56 -08:00
Ahmed Hussein	747883af4d	HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443 ). Contributed by Ahmed Hussein.	2020-11-20 18:03:45 +05:30
Masatake Iwasaki	425996eb4a	HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467 )	2020-11-18 16:11:09 +09:00
Akira Ajisaka	b57f04cd5b	HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465 )	2020-11-17 10:56:56 +09:00
Takanobu Asanuma	f56cd88d7d	HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.	2020-11-13 11:02:26 +09:00
Issac Buenrostro	f5e6be337b	HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255 ) Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>	2020-11-12 14:57:26 -08:00
Peter Bacsko	cd0490e8c6	Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." This reverts commit `b0ab222a6c`.	2020-11-11 17:06:12 +01:00
Peter Bacsko	b0ab222a6c	YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.	2020-11-11 17:03:16 +01:00
jianghuazhu	e6d2dccbef	HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138 )	2020-11-10 16:34:17 -08:00
Siyao Meng	fbd2220167	HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448 )	2020-11-10 12:13:15 -08:00
Yizhong Zhang	82de07c794	HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.	2020-11-10 09:11:31 -08:00
Hui Fei	f1d5652359	HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang.	2020-11-10 13:09:03 +08:00
maobaolong	95c96605b3	HDFS-15667. Audit log record the unexpected allowed result when delete (#2437 )	2020-11-10 13:01:10 +08:00
LeonGao	9a9ab5b48e	HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288 ). Contributed by Leon Gao.	2020-11-09 15:06:16 -08:00
He Xiaoqiao	3067a25fa1	HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Yiqun Lin <yqlin@apache.org>	2020-11-04 13:53:46 +08:00
Ayush Saxena	7ac7840fc3	HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424 ). Contributed by Ayush Saxena.	2020-11-04 09:28:44 +05:30
Konstantin V Shvachko	d07dc7afb4	HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.	2020-11-03 12:02:40 -08:00
Ahmed Hussein	324879127a	HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419 )	2020-10-28 16:24:34 -07:00
Akira Ajisaka	b3ba74d72d	HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418 )	2020-10-28 19:12:23 +09:00
Ahmed Hussein	d0c786db4d	HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406 )	2020-10-27 18:56:40 -07:00
Ahmed Hussein	98097b8f19	HDFS-15457. TestFsDatasetImpl fails intermittently (#2407 )	2020-10-27 18:52:56 -07:00
Hui Fei	8a6d5b9151	HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416 )	2020-10-28 09:13:25 +08:00
Ahmed Hussein	ae74407ac4	HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 20:45:00 +09:00
Akira Ajisaka	3848b44b37	HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309 )	2020-10-27 19:18:08 +09:00
Ahmed Hussein	afaab3d332	HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 16:56:30 +09:00
Hui Fei	12c908c827	HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.	2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang	74634eb002	HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.	2020-10-23 12:05:53 -07:00
Inigo Goiri	e7aa4da0a2	HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.	2020-10-23 09:47:10 -07:00
Hui Fei	264c948e60	HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang	2020-10-23 14:06:22 +08:00
Kihwal Lee	da1b6e3cc2	HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.	2020-10-22 16:59:49 -05:00
Kihwal Lee	88a9f42f32	HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.	2020-10-21 00:59:35 -05:00
Takanobu Asanuma	30f06e0c74	HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394 )	2020-10-20 19:08:59 +09:00
Ayush Saxena	2e8cafac3b	HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.	2020-10-19 10:48:47 +05:30
Ayush Saxena	173310e2f5	HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.	2020-10-19 10:34:17 +05:30
Ayush Saxena	1e3a6efcef	HADOOP-17288. Use shaded guava from thirdparty. (#2342 ). Contributed by Ayush Saxena.	2020-10-17 12:01:18 +05:30
Kihwal Lee	740a2c4635	HDFS-15627. Log delete audits before collecting blocks.	2020-10-16 11:13:41 -05:00
Ayush Saxena	cc57eebe45	HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386 ). Contributed by Ayush Saxena.	2020-10-16 10:36:24 +05:30
Kihwal Lee	e45407128d	HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein.	2020-10-14 17:26:04 -05:00
Siyao Meng	a308a1ec22	HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370 )	2020-10-13 10:59:42 -07:00
Gabor Bota	59f01a548e	HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same Contributed by Gabor Bota.	2020-10-13 16:17:44 +01:00
Mukund Thakur	0507c4160f	HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur. Change-Id: I6cec41c20467920cf21f169653553535414b2680	2020-10-13 13:31:00 +01:00
Steve Loughran	d606ddc12c	Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 )" (reverting to change the JIRA number) This reverts commit `3e1b1dfd77`. Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd	2020-10-13 13:30:38 +01:00
Mukund Thakur	3e1b1dfd77	HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur.	2020-10-13 13:27:55 +01:00
Konstantin V Shvachko	b3786d6c3c	HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.	2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G	b76b36ebbc	HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378 ). Contributed by Uma Maheswara Rao G.	2020-10-12 14:22:44 -07:00
Akira Ajisaka	69ef9b1ee8	HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375 )	2020-10-12 09:52:12 +09:00
LeonGao91	de055ec8aa	HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299 )	2020-10-10 22:08:36 +05:30
Hui Fei	518a212cff	HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363 )	2020-10-09 16:12:22 +08:00
Karthik Palanisamy	735e85a1d2	HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365 )	2020-10-08 12:51:04 +05:30
Karthik Palanisamy	df4006eb81	HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366 )	2020-10-07 22:09:04 +05:30
bshashikant	16aea11c94	HDFS-15611. Add list Snapshot command in WebHDFS. (#2355 )	2020-10-07 10:34:32 +05:30
Akira Ajisaka	074f0d46af	HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360 ) * Support getHAServiceState in DFSRouter	2020-10-07 13:37:45 +09:00
hemanthboyina	921ca1f554	HDFS-15543. RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled. Contributed by Hemanth Boyina.	2020-10-07 09:58:53 +05:30
Doroszlai, Attila	1cfe5916e2	HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-10-06 14:58:42 -07:00
Fei Hui	6ece6407e1	HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356 ). Contributed by Hui Fei.	2020-10-05 22:11:24 +05:30
Siyao Meng	43b0c0b054	HDFS-15607. Create trash dir when allowing snapshottable dir (#2352 )	2020-10-05 05:02:00 -07:00
huangtianhua	a89ca56a1b	HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347 )	2020-09-29 19:00:55 -07:00
Ayush Saxena	d686a655bc	HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui.	2020-09-29 00:03:11 +05:30
Ayush Saxena	fc8a6dd8a9	HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui.	2020-09-28 23:55:31 +05:30
Fei Hui	984875c624	HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345 )	2020-09-28 09:54:01 -07:00
Inigo Goiri	eae0035a2d	HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao.	2020-09-28 09:40:39 -07:00
Takanobu Asanuma	d89af79507	HDFS-15577. Refactor TestTracing. (#2302 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2020-09-28 18:27:41 +09:00
Vinayakumar B	82b86e3754	HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng	2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G	899dea2a21	HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-25 21:21:01 -07:00
Ye Ni	00c4de63cf	HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332 )	2020-09-25 09:47:54 -07:00
Uma Maheswara Rao G	3ccc962b99	HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-24 07:07:48 -07:00
Aryan Gupta	486ddb73f9	HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2020-09-24 21:15:55 +09:00
huangtianhua	ff59fbb8b0	HDFS-15025. Applying NVDIMM storage media to HDFS (#2189 ) * HDFS-15025. Applying NVDIMM storage media to HDFS Co-authored-by: YaYun-Wang <yywangyayun@163.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>	2020-09-24 14:27:04 +05:30
bshashikant	368f2f637e	HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326 )	2020-09-24 14:00:41 +05:30
Ye Ni	ae089f2db7	HDFS-15557. Log the reason why a storage log file can't be deleted (#2274 )	2020-09-22 13:23:20 -07:00
Hui Fei	474fa80bfb	HADOOP-17277. Correct spelling errors for separator (#2322 ) Contributed by Hui Fei.	2020-09-22 17:10:33 +01:00
Kihwal Lee	dfc2682213	HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross.	2020-09-22 10:53:04 -05:00
lfengnan	3e8b1e7426	HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266 ). Contributed by Fengnan Li.	2020-09-21 23:24:13 +05:30
Ayush Saxena	7a6265ac42	HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang.	2020-09-21 08:50:08 +05:30
Ayush Saxena	8eba922bb3	HDFS-15579. RBF: The constructor of PathLocation may got some misunderstanding. Contributed by Janus Chow.	2020-09-20 08:54:25 +05:30
Ayush Saxena	2377df38ad	HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.	2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G	7bba4c609c	HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312 ). Contributed by Uma Maheswara Rao G.	2020-09-18 15:18:10 +05:30
S O'Donnell	20a0e6278d	HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell	2020-09-17 14:11:42 +01:00
bshashikant	425f48799c	HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296 )	2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G	e4cb0d3514	HDFS-15578: Fix the rename issues with fallback fs enabled (#2305 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-16 22:43:00 -07:00
Takanobu Asanuma	5c5b2ed7c7	HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.	2020-09-16 01:36:46 +09:00
hemanthboyina	f4ed9f3f91	HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.	2020-09-15 10:10:21 +05:30
He Xiaoqiao	924959088e	HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.	2020-09-15 12:05:04 +08:00
Siyao Meng	6d3312de47	HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258 )	2020-09-14 13:31:34 -07:00
Akira Ajisaka	c78d18023d	HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267 )	2020-09-14 11:34:52 +09:00
Uma Maheswara Rao G	d2779de3f5	HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298 ). Contributed by Uma Maheswara Rao G.	2020-09-12 17:06:39 -07:00
Mingliang Liu	f59f7f2175	HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell	2020-09-12 01:41:38 -07:00
hemanthboyina	2a6fdcedeb	HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.	2020-09-11 18:21:25 +05:30
imbajin	89428f142f	HDFS-15551. Tiny Improve for DeadNode detector (#2265 ) Contributed by imbajin. Reviewed-by: leosunli <lisheng.sun08@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-09-11 15:44:03 +08:00
1996fanrui	56ebabd426	HADOOP-17222. Create socket address leveraging URI cache (#2241 ) Contributed by fanrui. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-09-10 22:30:52 -07:00
zz	12a316cdf9	HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185 ). Contributed by Zhenzhao Wang. Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>	2020-09-10 21:20:32 -07:00
Siyao Meng	43c52d6495	HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir (#2295 ) Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc	2020-09-10 15:55:23 +05:30
He Xiaoqiao	1d6d0d8207	HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun. Co-authored-by: Chen Zhang <chzhang1987@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2020-09-09 20:09:13 +08:00
Uma Maheswara Rao G	ac7d4623ae	HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275 ) Contributed by Uma Maheswara Rao G.	2020-09-07 11:36:13 -07:00
Uma Maheswara Rao G	b3660d0147	HDFS-15529: getChildFilesystems should include fallback fs as well (#2234 ). Contributed by Uma Maheswara Rao G.	2020-09-03 11:06:20 -07:00
He Xiaoqiao	60de592a88	HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.	2020-08-31 14:49:12 +08:00
bshashikant	2d03209a06	HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion (#2251 )	2020-08-31 11:29:48 +05:30
Steve Loughran	c392d9022a	HDFS-15471. TestHDFSContractMultipartUploader failing (#2252 ) Contributed by Steve Loughran (Was: broken by Steve Loughran)	2020-08-28 15:39:09 +01:00
Vivek Ratnavel Subramanian	06793da100	HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory (#2230 )	2020-08-27 11:36:30 -07:00
Tsz-Wo Nicholas Sze	41182a9b6d	HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. (#2233 )	2020-08-27 14:54:52 +05:30
Stephen O'Donnell	2ffe00fc46	HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-08-26 23:05:32 -07:00
Takanobu Asanuma	ca8e7a7725	HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina.	2020-08-27 12:10:39 +09:00
hemanthboyina	6e618b6a7e	HDFS-15536. RBF: Clear Quota in Router was not consistent.	2020-08-26 13:03:08 +05:30
S O'Donnell	6a49bf9bff	HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.	2020-08-25 15:18:36 +01:00
He Xiaoqiao	a7830423c5	HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.	2020-08-24 21:16:00 +08:00
hemanthboyina	83fd15b412	HDFS-14504. Rename with Snapshots does not honor quota limit.	2020-08-23 16:10:47 +05:30
Chen Liang	f734455e5d	HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.	2020-08-20 15:25:10 -07:00
Ayush Saxena	4813a37023	HDFS-15535. RBF: Fix Namespace path to snapshot path resolution for snapshot API. Contributed by Ayush Saxena.	2020-08-20 23:54:35 +05:30
Uma Maheswara Rao G	dd013f2fdf	HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229 ). Contributed by Uma Maheswara Rao G.	2020-08-19 09:30:41 -07:00
Siyao Meng	b65e43fe38	HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS (#2220 )	2020-08-18 19:28:19 +09:00
bshashikant	092bfe7c8e	HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. (#2172 ) * HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. * Addressed review comments. * Rebased to latest trunk and added snapshotTrashRoot config check. * Addressed review comments. * Removede empty line added in SnapshotManager.Java. * Addressed whitespace issues.	2020-08-17 12:56:13 +05:30
Ayush Saxena	e3d1966f58	HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.	2020-08-15 15:07:58 +05:30
Tsz-Wo Nicholas Sze	b93dd7c281	HDFS-15519. Check inaccessible INodes in FsImageValidation. (#2224 )	2020-08-14 10:10:01 -07:00
bshashikant	15a76e8d65	HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. (#2219 )	2020-08-14 20:46:48 +05:30
Vivek Ratnavel Subramanian	cb50e3fcf7	HDFS-15496. Add UI for deleted snapshots (#2212 )	2020-08-13 22:36:15 +05:30
Aryan Gupta	e592ec5f8b	HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. (#2217 ) * HDFS-15518. Fixed String operationName = ListSnapshot. * HDFS-15518. Changed ListSnapshot to operationName. Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>	2020-08-13 00:09:38 +05:30
S O'Donnell	10716040a8	HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang	2020-08-12 09:02:47 +01:00
Tsz-Wo Nicholas Sze	11cec9ab94	HDFS-15523. Fix findbugs warnings from HDFS-15520. (#2218 )	2020-08-11 21:41:28 -07:00
Siyao Meng	3fd3aeb621	HDFS-15492. Make trash root inside each snapshottable directory (#2176 )	2020-08-11 08:52:16 -07:00
Gautham B A	909f1e82d3	HADOOP-17196. Fix C/C++ standard warnings (#2208 ) * Passing C/C++ standard flags -std is not cross-compiler friendly as not all compilers support all values. * Thus, we need to make use of the appropriate flags provided by CMake in order to specify the C/C++ standards. Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-08-11 16:35:08 +09:00
Uma Maheswara Rao G	8955a6ceb7	HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205 ) * HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false * Used LambdaTestUtils#intercept in test	2020-08-11 00:01:58 -07:00
Tsz-Wo Nicholas Sze	592127bdc2	HDFS-15520 Use visitor pattern to visit namespace tree (#2203 )	2020-08-11 12:30:30 +05:30
Akira Ajisaka	32895f4f7e	HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.	2020-08-11 13:49:56 +09:00
Akira Ajisaka	7938ebfb9d	HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.	2020-08-11 11:14:58 +09:00
Akira Ajisaka	9062814bab	HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module	2020-08-11 10:58:56 +09:00
Ayush Saxena	c2a17659d1	HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.	2020-08-08 14:33:53 +05:30
Ayush Saxena	975b6024dd	HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.	2020-08-07 22:19:17 +05:30
Takanobu Asanuma	1d5ccc790b	HDFS-15512. Remove smallBufferSize in DFSClient. (#2191 )	2020-08-06 22:47:12 +05:30
Mingliang Liu	58def7cecb	HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-08-04 20:48:45 -07:00
bshashikant	e072d33327	HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable (#2175 )	2020-08-04 14:10:29 +05:30
hemanthboyina	82f3ffcd64	HDFS-15503. File and directory permissions are not able to be modified from WebUI.	2020-08-03 23:21:01 +05:30
Akira Ajisaka	c40cbc57fa	HADOOP-17091. [JDK11] Fix Javadoc errors (#2098 )	2020-08-03 10:46:51 +09:00
bshashikant	d8a2df25ad	HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181 )	2020-08-02 21:59:56 +05:30
hemanthboyina	528a799a78	HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.	2020-08-02 12:02:37 +05:30
Yuan	e756fe3590	HDFS-14950. fix missing libhdfspp lib in dist-package (#1947 ) libhdfspp.{a,so} are missed in dist-package. This patch fixed this by copying these libs to the right directory Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>	2020-07-31 16:49:49 +09:00
Tsz-Wo Nicholas Sze	05b3337a46	HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots (#2165 )	2020-07-30 23:06:51 +05:30
bshashikant	68287371cc	HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166 )	2020-07-29 21:33:25 +05:30
touchida	026dce5334	HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. (#2135 )	2020-07-27 09:55:11 -07:00
bshashikant	2d12496643	HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (#2163 )	2020-07-22 16:16:27 -07:00
Uma Maheswara Rao G	ac9a07b51a	HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160 ). Contributed by Uma Maheswara Rao G.	2020-07-21 23:29:10 -07:00
Masatake Iwasaki	1b29c9bfee	HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155 )	2020-07-22 13:40:20 +09:00
Inigo Goiri	8b7695bb26	HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.	2020-07-21 09:00:07 -07:00
Tsz-Wo Nicholas Sze	d57462f2da	HDFS-15479. Ordered snapshot deletion: make it a configurable feature (#2156 )	2020-07-21 11:36:24 +05:30
Shashikant Banerjee	d9441f95c3	HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.	2020-07-21 11:13:05 +05:30
Chen Liang	3833c616e0	HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.	2020-07-20 12:49:58 -07:00
Ayush Saxena	6cbd8854ee	HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.	2020-07-20 21:49:05 +05:30
Tsz-Wo Nicholas Sze	2cec50cf16	HDFS-15463. Add a tool to validate FsImage. (#2140 )	2020-07-20 11:44:30 +05:30
Suraj Naik	4101b0c0ed	HDFS-15476 Make AsyncStream executor private (#2151 )	2020-07-19 23:26:54 +05:30
Ayush Saxena	8a9a674ef1	HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu.	2020-07-18 15:36:55 +05:30
Ayush Saxena	6bcb24d269	HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.	2020-07-18 14:33:43 +05:30
Ahmed Hussein	1f71c4ae71	HADOOP-17099. Replace Guava Predicate with Java8+ Predicate Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 11:39:06 -05:00
Ahmed Hussein	98fcffe93f	HADOOP-17101. Replace Guava Function with Java8+ Function Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 09:53:18 -05:00
Shashikant Banerjee	85d4718ed7	HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.	2020-07-15 10:32:25 +05:30
Gautham B A	cce5a6f609	HDFS-15385 Upgrade boost library to 1.72 (#2051 ) * Removed the asio-1.10.2 that was checked in under libhdfspp/third_party directory.	2020-07-15 13:39:48 +09:00
zhaoyim	bdce75d737	HDFS-15371. Nonstandard characters exist in NameNode.java (#2032 ) Contributed by zhaoyim	2020-07-14 14:42:12 +01:00
Steve Loughran	b9fa5e0182	HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. Contributed by Steve Loughran. Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3	2020-07-13 13:30:02 +01:00
He Xiaoqiao	b97fea65e7	HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.	2020-07-13 14:12:48 +08:00
lfengnan	84b74b335c	HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110 )	2020-07-12 00:10:12 -07:00
Uma Maheswara Rao G	3e70006639	HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132 ). Contributed by Uma Maheswara Rao G.	2020-07-11 23:50:04 -07:00
Xiaoyu Yao	f91a8ad88b	HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085 )	2020-07-09 11:33:37 -07:00
He Xiaoqiao	4f26454a7d	HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang.	2020-07-08 00:02:39 +08:00
Ye Ni	f77bbc2123	HDFS-15312. Apply umask when creating directory by WebHDFS (#2096 )	2020-07-06 19:01:46 -07:00
Uma Maheswara Rao G	dc0626b5f2	HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.	2020-07-06 18:50:03 -07:00
Ye Ni	e820baa6e6	HDFS-15417. RBF: Get the datanode report from cache for federation WebHDFS operations (#2080 )	2020-07-06 16:17:09 -07:00
Shanyu Zhao	834372f404	HDFS-15451. Do not discard non-initial block report for provided storage. (#2119 ). Contributed by Shanyu Zhao. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-07-06 23:43:34 +08:00
Uma Maheswara Rao G	55a2ae80dc	HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.	2020-07-04 13:45:49 -07:00
Ayush Saxena	f86f15cf20	HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.	2020-07-04 13:00:43 +05:30
Uma Maheswara Rao G	1f2a80b5e5	HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-07-04 00:12:10 -07:00
He Xiaoqiao	9ac498e300	HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.	2020-07-01 12:38:07 +08:00
Stephen O'Donnell	2a67e2b1a0	HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-30 07:10:17 -07:00
Akira Ajisaka	c71ce7ac33	HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.	2020-06-28 16:02:47 +09:00
Ayush Saxena	8db38c98a6	HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.	2020-06-27 19:13:01 +05:30
Virajith Jalaparti	bed0a3a374	HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100 ) * HDFS-15436. Default mount table name used by ViewFileSystem should be configurable * Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests * Address Uma's comments on PR#2100 * Sort lists in test to match without concern to order * Address comments, fix checkstyle and fix failing tests * Fix checkstyle	2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G	d5e1bb6155	HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-06-26 01:29:38 -07:00
lfengnan	84110d850e	HDFS-15383. RBF: Add support for router delegation token without watch (#2047 ) Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.	2020-06-23 13:12:29 -07:00
Uma Maheswara Rao G	7c02d1889b	HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.	2020-06-23 01:42:25 -07:00
Ayush Saxena	17ffcab5f6	HDFS-14546. Document block placement policies. Contributed by Amithsha.	2020-06-21 17:27:28 +05:30
Uma Maheswara Rao G	b27810aa60	HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.	2020-06-20 00:32:02 -07:00
Stephen O'Donnell	d50e93ce7b	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang	edf716a5c3	Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." This reverts commit `730a39d138`.	2020-06-18 06:45:15 -07:00
S O'Donnell	123777823e	HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina	2020-06-18 12:26:22 +01:00
Yiqun Lin	9cbd76cc77	HDFS-15346. FedBalance tool implementation. Contributed by Jinglun.	2020-06-18 13:33:25 +08:00
Ayush Saxena	c8ed33cd2a	HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.	2020-06-17 13:55:40 +05:30
Stephen O'Donnell	730a39d138	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-16 15:59:13 -07:00
Eric Yang	81d8a887b0	SPNEGO TLS verification Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-06-15 10:56:16 +09:00
Takanobu Asanuma	f41a144077	HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.	2020-06-15 09:15:53 +09:00
Inigo Goiri	719b53a79d	HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.	2020-06-13 09:35:05 -07:00
Uma Maheswara Rao G	785b1def95	HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.	2020-06-12 14:32:19 -07:00
Vinayakumar B	e154084770	HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026 )	2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G	93b121a971	HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.	2020-06-10 15:00:02 -07:00
Ayush Saxena	b735a77717	HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.	2020-06-10 12:06:16 +05:30
Ayush Saxena	635e6a16d0	HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi.	2020-06-10 11:50:37 +05:30

... 8 9 10 11 12 ...

9790 Commits