hadoop

Author	SHA1	Message	Date
Stephen O'Donnell	34a973a90e	HDFS-16610. Make fsck read timeout configurable (#4384 )	2022-06-01 20:36:01 +01:00
Gautham B A	4496e2daf6	HDFS-16604. Install gtest via FetchContent_Declare in CMake (#4374 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-06-01 13:45:44 +09:00
slfan1989	e85f827f0a	HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372 ) Co-authored-by: slfan1989 <louj1988@@>	2022-06-01 06:45:07 +08:00
Masatake Iwasaki	3228142f53	Make upstream aware of 2.10.2 release. (cherry picked from commit 3dcb6367edd44c04a5ae0d195d851abac80d10a0) Conflicts: hadoop-project-dist/pom.xml	2022-06-01 00:53:01 +09:00
slfan1989	5952934ad5	HDFS-16599. Fix typo in hadoop-hdfs-rbf module (#4368 ). Contributed by fanshilun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-05-31 01:59:17 +05:30
slfan1989	995f7b0956	HDFS-16597. RBF: Improve RouterRpcServer#reload With Lambda (#4362 ). Contributed by fanshilun. Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-05-28 09:48:34 +05:30
xuzq	dc9b21feac	HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339 ). Contributed by ZanderXu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-05-28 09:43:37 +05:30
Ayush Saxena	96985f4c45	HDFS-15225. RBF: Add snapshot counts to content summary in router. (#4356 ). Contributed by Ayush Saxena. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2022-05-27 09:06:07 +05:30
Ashutosh Gupta	bee538f785	HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337 ) Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com> Reviewed-by: Tao Li <tomscut@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-05-27 05:18:15 +08:00
Stephen O'Donnell	297f0f6d6a	HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332 ) Co-authored-by: S O'Donnell <sodonnell@cloudera.com>	2022-05-27 01:11:10 +08:00
Rishabh Sharma	0be1fde962	HDFS-16561. Handle error returned by strtol * strtol is used in hdfs-chmod.cc. The call to strtol could error out when an invalid input is provided. * This PR handles the error given out by strtol.	2022-05-26 11:09:49 +05:30
slfan1989	f469b0e143	HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335 ) * HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method() HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri() TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status(). * HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. * HADOOP-18249. Fix Some CheckStyle. Co-authored-by: slfan1989 <louj1988@@>	2022-05-24 23:11:10 +05:30
Michael Stack	5ab747222b	HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338 ) * HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit' Remove the ThreadGroup used by executor factories; they are unused and ThreadGroups auto-destroy when their Thread-member count goes to zero. This behavior is incompatible with the configuration we have on the per-volume executor which is set to let all threads die if no use inside the keepalive time. Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>	2022-05-23 08:45:08 -07:00
jianghuazhu	4fd4208439	HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333 ). Contributed by JiangHua Zhu. Signed-off-by: litao <tomleescut@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-05-22 21:59:38 +08:00
Viraj Jasani	93a13202d7	HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323 ) Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Tao Li <tomscut@apache.org>	2022-05-21 09:28:59 +08:00
Viraj Jasani	0733e968ab	HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 (#4267 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-05-21 03:21:13 +09:00
Ashutosh Gupta	fb910bd906	HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229 ) Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-05-21 02:53:14 +09:00
Steve Loughran	6f261ed4a2	HADOOP-18198. Release 3.3.3: release notes and jdiff files. * Add the changelog and release notes * add all jdiff XML files * update the project pom with the new stable version Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d	2022-05-17 19:00:54 +01:00
Hui Fei	dad2900b87	Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199 )" (#4306 )	2022-05-13 14:44:41 +08:00
Felix Nguyen	680af27aa6	HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199 )	2022-05-13 14:28:53 +08:00
singer-bin	d74f328b00	HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122 )	2022-05-12 16:53:09 +08:00
Gautham B A	1350539f2d	HDFS-16465. Remove redundant strings.h inclusions (#4279 ) * Remove redundant strings.h inclusions * strings.h was included in a bunch of C/C++ files and were redundant. * Also, strings.h is not available on Windows and thus isn't cross-platform compatible. * Build for all platforms in CI * Revert "Build for all platforms in CI" This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb. * Debug failure on Centos 8 * Skipping pipeline run on Centos 7 to debug the failure on Centos 8. * Revert "Debug failure on Centos 8" This reverts commit e365e34d6fab9df88f4df622910ddb28a8c8796f.	2022-05-11 23:04:22 +05:30
Viraj Jasani	6983aef945	HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264 ) Signed-off-by: Tao Li <tomscut@apache.org>	2022-05-11 08:10:03 +08:00
daimin	d486ae8c0f	Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291 )	2022-05-10 08:01:16 +08:00
Ashutosh Gupta	01027e52a9	HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-05-08 08:05:34 +09:00
Ashutosh Gupta	ae47846a5b	HADOOP-16515. Update the link to compatibility guide (#4226 ) Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-05-08 07:38:32 +09:00
daimin	29401c8203	HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104 ) Reviewed-by: Hui Fei <ferhui@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-05-07 01:30:32 +08:00
Gautham B A	d346be9bd2	HDFS-16564. Use uint32_t for hdfs_find (#4245 ) * hdfs_find uses u_int32_t type for storing the value for the max-depth command line argument. * The type u_int32_t isn't standard, isn't available on Windows and thus breaks cross-platform compatibility. We need to replace this with uint32_t which is available on all platforms since it's part of the C++ standard.	2022-05-04 20:19:08 +05:30
Murali Krishna	9807dfd25c	HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240 )	2022-05-03 17:42:45 +05:30
Viraj Jasani	2dfa928a20	HDFS-16521. DFS API to retrieve slow datanodes (#4107 ) Signed-off-by: stack <stack@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-05-03 05:05:40 +08:00
Gautham B A	88155cebe9	HDFS-16468. Define ssize_t for Windows (#4228 ) * Some C/C++ files use ssize_t data type. This isn't available for Windows and we need to define an alias for this and set it to an appropriate type to make it cross platform compatible.	2022-04-29 22:25:09 +05:30
smarthan	7bd7725532	HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211 ). Contributed by Chengwei Wang.	2022-04-29 22:59:23 +08:00
Viraj Jasani	b9ade7a4cd	HDFS-16528. Reconfigure slow peer enable for Namenode (#4186 )	2022-04-29 17:00:17 +08:00
stack	9ed8d60511	HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170 ) This reverts the previous commit `4e47eb66d1` undone so I could reapply with the '.' after the HDFS-16540 as is done in all other commits.	2022-04-28 10:28:01 -07:00
stack	4e47eb66d1	Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170 )" Revert to add the '.' after HDFS-16540 so commit message format matches precedent This reverts commit `bda0881bf9`.	2022-04-28 10:26:46 -07:00
huaxiangsun	bda0881bf9	HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170 ) When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.	2022-04-28 10:24:37 -07:00
Felix Nguyen	b4ff49a394	HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168 )	2022-04-27 14:42:30 +08:00
smarthan	f187e9bcd5	HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213 ). Contributed by Chengwei Wang.	2022-04-27 11:44:38 +08:00
litao	f1e5f8e764	HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234 )	2022-04-26 11:48:06 +08:00
Ashutosh Gupta	89279f0b90	HDFS-16255. RBF: Fix dead link to fedbalance document (#4193 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-04-25 14:11:41 +09:00
Hui Fei	17d64ba495	Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035 )" (#4232 )	2022-04-24 17:18:31 +08:00
litao	acc0e0a210	HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035 )	2022-04-24 14:54:41 +08:00
daimin	aebd55f788	HDFS-16519. Add throttler to EC reconstruction (#4101 ) Reviewed-by: litao <tomleescut@gmail.com> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2022-04-23 12:45:22 +09:00
litao	5ebbacc480	HDFS-16552. Fix NPE for TestBlockManager (#4210 )	2022-04-23 12:14:06 +09:00
smarthan	bf4730b4d6	HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. (#4061 ). Contributed by Chengwei Wang.	2022-04-21 12:32:12 +08:00
S O'Donnell	a4683be65e	Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148 ). Contributed by Stephen O'Donnell." This reverts commit `dbeeee0363`.	2022-04-20 20:34:43 +01:00
qinyuren	76bbd17374	HDFS-16544. EC decoding failed due to invalid buffer (#4179 )	2022-04-20 15:04:27 +09:00
Ashutosh Gupta	ec0ff1dc04	HDFS-16035. Remove DummyGroupMapping as it is not longer used anywhere. (#4183 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-04-19 14:35:23 +09:00
qinyuren	52e152f8b0	HDFS-16538. EC decoding failed due to not enough valid inputs (#4167 ) Co-authored-by: liubingxing <liubingxing@bigo.sg>	2022-04-19 13:37:28 +09:00
Happy-shi	f74f241636	HDFS-16541. Fix a typo in NameNodeLayoutVersion (#4176 )	2022-04-18 22:53:57 +05:30
jianghuazhu	900682e712	HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-04-18 11:21:51 +08:00
Quanlong Huang	35d4c02bcc	HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158 ) Reviewed-by: Lisheng Sun <sunlisheng@apache.org>	2022-04-18 10:32:29 +08:00
Stephen O'Donnell	dbeeee0363	HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148 ). Contributed by Stephen O'Donnell.	2022-04-17 20:05:11 +08:00
He Xiaoqiao	cb975c3df6	HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141 ) Contributed by limingxiang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-04-17 19:21:27 +08:00
Renukaprasad C	f14f305051	HDFS-16526. Add metrics for slow DataNode (#4162 )	2022-04-15 21:37:05 +05:30
caozhiqiang	cee8c62498	HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126 )	2022-04-14 18:42:39 +09:00
daimin	c65c383b7e	HDFS-16509. Fix decommission UnsupportedOperationException (#4077 ). Contributed by daimin.	2022-04-14 11:07:06 +08:00
Takanobu Asanuma	2efab92959	HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138 )	2022-04-14 11:23:38 +09:00
Steve Loughran	a7b4e8f03e	HDFS-14478. Add libhdfs APIs for openFile (#4166 ) Contributed by Sahil Takiar	2022-04-13 14:15:27 +01:00
qinyuren	45394433a1	HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032 )	2022-04-13 11:27:37 +09:00
Gautham B A	e89fd9645b	HDFS-16474. Make HDFS tail tool cross platform (#4157 ) * The source files for hdfs_tail uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make these tools cross platform.	2022-04-12 20:31:24 +05:30
Gautham B A	d5e97fe4d6	HDFS-16473. Make HDFS stat tool cross platform (#4145 ) * The source files for hdfs_stat uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make this tool cross platform.	2022-04-08 23:06:39 +05:30
Takanobu Asanuma	37650ced81	HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang.	2022-04-08 18:38:40 +09:00
litao	5412fbf6d4	HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001 )	2022-04-08 12:26:39 +08:00
singer-bin	807a428b55	HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069 )	2022-04-08 10:27:35 +09:00
Viraj Jasani	f70935522b	HADOOP-18188. Support touch command for directory (#4135 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-04-07 17:29:45 +09:00
Viraj Jasani	7c20602b17	HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-04-06 18:17:02 +09:00
wangzhaohui	61bbdfd3a7	HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-04-06 17:29:43 +09:00
Stephen O'Donnell	bbfe3500cf	HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142 )	2022-04-06 08:49:52 +01:00
Gautham B A	4ef1d3eef9	HDFS-16472. Make HDFS setrep tool cross platform (#4130 ) * The source files for hdfs_setrep uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * We need to replace getopt with boost::program_options to make this tool cross platform.	2022-04-05 22:59:11 +05:30
litao	34b3275bf4	HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-04-03 01:36:03 +05:30
Xing Lin	15a5ea2c95	HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094 ) HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS	2022-03-31 21:59:09 +00:00
litao	9a4dddd640	HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082 )	2022-03-30 23:01:48 -07:00
He Xiaoqiao	2bf78e2416	HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mode. (#4085 ). Contributed by limingxiang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-03-31 14:00:38 +08:00
litao	ac50657c37	HDFS-16413. Reconfig dfs usage parameters for datanode (#3863 )	2022-03-31 00:51:17 +09:00
He Xiaoqiao	6eea28c3f3	HDFS-16498. Fix NPE for checkBlockReportLease #4057 . Contributed by tomscut. Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-03-30 14:56:04 +08:00
Lei Yang	a9b43966c0	HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache Fixes #4100 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>	2022-03-28 13:15:28 -07:00
Masatake Iwasaki	0fbd96a244	Make upstream aware of 3.2.3 release.	2022-03-28 08:07:50 +00:00
GuoPhilipse	046a6204b4	HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724 ) Co-authored-by: gf13871 <gf13871@ly.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-03-27 22:23:48 +09:00
litao	565e848d88	HDFS-16434. Add opname to read/write lock for remaining operations (#3915 )	2022-03-25 20:19:01 +09:00
qinyuren	45ce1cce50	HDFS-16501. Print the exception when reporting a bad block (#4062 ) Reviewed-by: tomscut <litao@bigo.sg>	2022-03-23 15:03:17 +09:00
He Xiaoqiao	ef8bff0df9	HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918 ). Contributed by Hongbing Wang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-03-23 11:32:12 +08:00
He Xiaoqiao	26ba3846cc	Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918 ). Contributed by Hongbing Wang." This reverts commit `88975496d8`.	2022-03-23 11:01:35 +08:00
Gautham B A	81879eb9cb	HDFS-16471. Make HDFS ls tool cross platform (#4086 ) * The source files for hdfs_ls uses getopt for parsing the command line arguments. * getopt is available only on Linux and thus, isn't cross platform. * Thus, we need to replace getopt with boost::program_options to make this tool cross platform.	2022-03-22 23:12:00 +05:30
He Xiaoqiao	88975496d8	HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918 ). Contributed by Hongbing Wang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-03-22 22:28:09 +08:00
Owen O'Malley	1d5650c4d0	HDFS-13248: Namenode needs to use the actual client IP when going through the RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures the list of users than can set their client ip address using the client context. Fixes #4081	2022-03-21 09:27:35 -07:00
litao	e5549a2a68	HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-03-21 12:40:51 +05:30
He Xiaoqiao	62d59e516e	HDFS-16504. Add parameter for NameNode to process getBloks request. (#4068 ). Contributed by Max Xie. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-03-20 14:01:45 +08:00
Gautham B A	a631f45a99	HDFS-16470. Make HDFS find tool cross platform (#4076 ) * The source files for hdfs_find uses getopt for parsing the command line arguments. getopt is available only on Linux and thus, isn't cross platform. * Thus, we need to replace getopt with boost::program_options to make hdfs_find cross platform.	2022-03-18 21:41:01 +05:30
litao	7f6a891f03	HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058 )	2022-03-17 12:04:48 +08:00
He Xiaoqiao	a237526988	HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048 ). Contributed by JiangHua Zhu.	2022-03-16 20:43:38 +08:00
Viraj Jasani	1c0bc35305	HDFS-16502. Reconfigure Block Invalidate limit (#4064 ) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-03-16 09:32:29 +08:00
Xing Lin	8b8158f02d	HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true. If the trash root for path p is in the same mount point as path p, and one of: * The mount point isn't at the top of the target fs. * The resolved path of path is root (eg it is the fallback FS). * The trash root isn't in user's target fs home directory. get the corresponding viewFS path for the trash root and return it. Otherwise, use <mnt>/.Trash/<user>. Signed-off-by: Owen O'Malley <oomalley@linkedin.com>	2022-03-14 11:29:48 -07:00
Owen O'Malley	7b5eac27ff	HDFS-16495: RBF should prepend the client ip rather than append it. Fixes #4054 Signed-off-by: Owen O'Malley <oomalley@linkedin.com>	2022-03-14 10:21:35 -07:00
He Xiaoqiao	a32cfc2169	HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941 ). Contributed by limingxiang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: litao <tomleescut@gmail.com>	2022-03-12 18:40:09 +08:00
Stephen O'Donnell	49e4e4d0a5	HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error (#4051 )	2022-03-08 11:07:11 +00:00
Viraj Jasani	66b72406bd	HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2022-03-08 17:27:04 +09:00
Gautham B A	8f07081789	HDFS-16462. Make HDFS get tool cross platform (#4003 )	2022-03-05 23:05:14 +05:30
Gautham B A	19561846df	HADOOP-18151. Switch the baseurl for Centos 8 (#4047 ) * Centos 8 has reached its End-of-Life and thus its packages are no longer accessible from mirror.centos.org. * This PR switches the baseurl to vault.centos.org where the packages are archived.	2022-03-05 07:11:22 +05:30
Viraj Jasani	278568203b	HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028 ). Contributed by Viraj Jasani.	2022-03-04 22:17:48 +05:30
Chao Sun	f800b65b40	Make upstream aware of 3.3.2 release	2022-03-02 19:14:50 -08:00
litao	9199787215	HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041 )	2022-03-02 11:25:50 +08:00
SevenAddSix	5eab9719cb	HDFS-16480. Fix typo: indicies -> indices (#4020 )	2022-02-28 21:12:18 +08:00
Hui Fei	b2f541a749	Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998 )" (#4038 )	2022-02-28 21:05:19 +08:00
litao	8aa568cea5	[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998 )	2022-02-28 21:03:59 +08:00
litao	6b07c851f3	HDFS-16397. Reconfig slow disk parameters for datanode (#3828 )	2022-02-25 01:23:20 +09:00
Viraj Jasani	e363f51ffb	HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002 )	2022-02-22 13:04:36 +09:00
jianghuazhu	589695c6a9	HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861 )	2022-02-22 10:15:19 +08:00
Neil	2f448acf39	HDFS-16459. RBF: register RBFMetrics in MetricsSystem for promethuessink (#3999 )	2022-02-21 11:32:34 -08:00
Chentao Yu	19d90e62fb	HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953)	2022-02-15 15:58:24 -08:00
YulongZ	48bef285a8	HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971 )	2022-02-15 08:44:17 -08:00
litao	0c194f2157	HDFS-16396. Reconfig slow peer parameters for datanode (#3827 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2022-02-15 13:40:46 +09:00
Neil	c777142ed3	RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. (#3965 )	2022-02-10 09:17:30 -08:00
Neil	390967f1f0	RBF: Add search box in Router's tab-mounttable web page (#3968 )	2022-02-09 09:17:23 -08:00
secfree	bf0cefb0d8	HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847 )	2022-02-09 10:07:05 +08:00
YulongZ	3e7a7c3c4a	HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878 )	2022-02-08 08:40:57 -08:00
singer-bin	125e3b6160	HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926 )	2022-02-06 13:05:57 +08:00
litao	b5b07af605	HDFS-16435. Remove no need TODO comment for ObserverReadProxyProvider (#3912 ). Contributed by tomscut. Reviewed-by: Chao Sun <sunchao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-02-03 08:21:06 +05:30
Gautham B A	ed44662968	HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945 )	2022-02-02 11:12:17 +05:30
KevinWikant	089e06de21	HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-01-31 13:04:32 +09:00
litao	bd50b9117e	HDFS-16444. Show start time of JournalNode on Web (#3943 )	2022-01-30 16:13:58 +08:00
secfree	39cad5f28f	HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: litao <tomleescut@gmail.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-01-28 17:04:11 +09:00
Li MingXiang	e17c96a40a	HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900 ). Contributed by limingxiang. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-01-27 16:53:21 +08:00
litao	6136d630a3	HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888 )	2022-01-27 13:42:44 +09:00
litao	c2ff39006f	HDFS-16398. Reconfig block report parameters for datanode (#3831 )	2022-01-26 17:24:09 +09:00
jianghuazhu	8c7c49d877	HDFS-16401.Remove the worthless DatasetVolumeChecker#numAsyncDatasetChecks. (#3838 )	2022-01-25 21:51:17 +08:00
Bryan Beaudreault	94b884ae55	HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527 )	2022-01-25 11:42:35 +00:00
Thinker313	43153e80cb	HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898 ). Contributed by lei w. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-01-25 15:26:18 +08:00
daimin	d69938994e	HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842 ) Reviewed-by: Istvan Fajth <pifta@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2022-01-25 13:02:37 +08:00
daimin	5ef335da1e	HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899 ). Contributed by daimin. Reviewed-by: tomscut <litao@bigo.sg> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2022-01-24 12:04:26 +05:30
litao	15b820c83c	HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. (#3839 ). Contributed by tomscut. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-01-24 14:26:30 +08:00
qinyuren	db2c3200e6	HDFS-16423. Balancer should not get blocks on stale storages (#3883 ) Reviewed-by: litao <tomleescut@gmail.com> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2022-01-19 17:59:42 +09:00
litao	e355646330	HDFS-16399. Reconfig cache report parameters for datanode (#3841 )	2022-01-19 15:10:39 +09:00
qinyuren	fcb1076699	HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887 )	2022-01-19 13:42:33 +09:00
Jackson Wang	d8862822d2	HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880 ) Reviewed-by: litao <tomleescut@gmail.com> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2022-01-14 22:38:11 +09:00
litao	f02374df92	HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2022-01-14 14:48:10 +09:00
Xiangyi Zhu	034dc8d03c	HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063 ). Contributed by Xiangyi Zhu. Reviewed-by: tomscut <litao@bigo.sg> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2022-01-13 12:56:19 +08:00
Gautham B A	271ac0e091	HDFS-16419. Make HDFS data transfer tools cross platform (#3873 )	2022-01-12 19:57:27 +05:30
luoyuan3471	e2d620192a	HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791 ) Co-authored-by: luoyuan <luoyuan@shopee.com>	2022-01-12 11:40:32 +08:00
Felix Nguyen	b88d5ae1c6	HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871 )	2022-01-10 18:28:20 +08:00
Viraj Jasani	f64fda0f00	HADOOP-18055. Async Profiler endpoint for Hadoop daemons (#3824 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2022-01-06 17:56:49 +08:00
litao	46b02788bd	HDFS-16371. Exclude slow disks when choosing volume (#3753 )	2022-01-06 09:31:00 +09:00
Ashutosh Gupta	43e5218a86	HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854 ) Contributed by Ashutosh Gupta	2022-01-05 15:51:14 +00:00
liever18	e1d0aa9ee5	HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856 )	2022-01-05 15:42:58 +00:00
Gautham B A	c3006be516	HDFS-16407. Make hdfs_du tool cross platform (#3848 )	2022-01-04 22:29:54 +05:30
Ashutosh Gupta	9eea0e28f2	HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2022-01-04 13:25:16 +09:00
Ayush Saxena	61d424f385	HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849 ). Contributed by Ayush Saxena. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2022-01-04 07:50:01 +05:30
Symious	b225287913	HDFS-16348. Mark slownode as badnode to recover pipeline (#3704 )	2021-12-30 00:27:11 +09:00
jianghuazhu	a5659fe0f7	HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-24 17:24:12 +09:00
KevinWikant	d20b598f97	HDFS-16303. Improve handling of datanode lost while decommissioning (#3675 ) Co-authored-by: Kevin Wikant <wikak@amazon.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-23 22:59:08 +09:00
secfree	cc07a5a186	HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821 )	2021-12-22 10:44:07 +08:00
wangzhaohui	8d251bd629	HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820 )	2021-12-21 11:28:57 +00:00
Jackson Wang	9420cb6f8a	HDFS-16385. Fix Datanode retrieve slownode information bug. (#3803 ) Reviewed-by: Symious <yiyang0203@gmail.com> Reviewed-by: Hui Fei <ferhui@apache.org> Reviewed-by: litao <tomleescut@gmail.com>	2021-12-21 11:04:02 +09:00
jianghuazhu	746b328554	HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806 )	2021-12-20 11:28:55 +00:00
secfree	32a78e0b9e	HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815 )	2021-12-20 18:16:22 +08:00
Viraj Jasani	04b6b9a87b	HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-20 16:01:34 +09:00
liubingxing	d8dea6f52a	HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714 ). Contributed by liubingxing. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-12-17 13:31:28 +08:00
Wei-Chiu Chuang	bdec546671	Revert "HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798 )" This reverts commit `a4557f9ed9`.	2021-12-16 21:27:08 +08:00
litao	22f5e1885d	HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-12-16 13:49:50 +09:00
Tamás Pénzes	a4557f9ed9	HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798 )	2021-12-16 12:38:42 +08:00
litao	10cdde59b4	HDFS-16375. The FBR lease ID should be exposed to the log (#3769 )	2021-12-16 13:29:32 +09:00
litao	59c650802d	HDFS-16378. Add datanode address to BlockReportLeaseManager logs (#3786 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-12-15 11:16:32 +08:00
Ayush Saxena	d29f0e83a9	HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: litao <tomleescut@gmail.com> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-12-14 17:21:51 +09:00
litao	c56a07f36b	HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716 )	2021-12-14 11:19:06 +09:00
Akira Ajisaka	9b9e2ef87f	HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774 ) Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>	2021-12-10 01:36:31 +09:00
Gautham B A	6dddbd42ed	HDFS-16285. Make HDFS ownership tools cross platform (#3588 )	2021-12-09 10:17:41 +05:30
qinyuren	35556ea043	HDFS-16333. fix balancer bug when transfer an EC block (#3679 )	2021-12-09 13:24:09 +09:00
PHILO-HE	1364847fdd	HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761 )	2021-12-08 17:52:40 +05:30
GuoPhilipse	00d483d6de	HDFS-16324. Fix error log in BlockManagerSafeMode (#3661 ) Reviewed-by: tomscut <litao@bigo.sg> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-08 21:21:48 +09:00
lzx404243	7ba91c1235	HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-08 20:44:18 +09:00
GuoPhilipse	25849ff7b7	HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696 ). Contributed by guophilipse. Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-08 16:34:54 +05:30
Masatake Iwasaki	e9526042a2	HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740 )	2021-12-07 21:39:04 +09:00
GuoPhilipse	80cccc5376	HDFS-16351. Add path exception information in FSNamesystem (#3713 ). Contributed by guophilipse. Reviewed-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-07 14:19:27 +05:30
GuoPhilipse	1509ea5de2	HDFS-16338. Correct fsimage error configuration message (#3684 ). Contributed by guophilipse. Reviewed-by: cxorm <lianp964@gmail.com> Reviewed-by: tomscut <litao@bigo.sg> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-06 14:00:50 +05:30
Takanobu Asanuma	e8e69de106	HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu.	2021-12-06 10:44:36 +09:00
litao	c2afb6a00b	HDFS-16370. Fix assert message for BlockInfo (#3747 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-04 22:54:44 +05:30
GuoPhilipse	2f55a6cced	HDFS-16357. Fix log format in DFSUtilClient (#3729 ). Contributed by guophilipse. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-04 18:35:05 +05:30
Ayush Saxena	cab7086fbc	HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745 ). Contributed by Ayush Saxena. Reviewed-by: litao <tomleescut@gmail.com> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-12-04 10:54:29 +05:30
huhaiyang	c0f405a46b	HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-04 02:19:36 +09:00
bitterfox	dd6b987c93	HDFS-16332. Handle invalid token exception in sasl handshake (#3677 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-12-03 23:30:13 +09:00
wangzhaohui	0cb6c28d19	HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 (#3742 )	2021-12-03 16:51:04 +05:30
litao	52ec65fd10	HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-12-03 14:12:05 +09:00
Viraj Jasani	0c62a514f7	HDFS-16358. HttpFS implementation for getSnapshotDiffReportListing (#3730 )	2021-12-03 13:25:32 +09:00
litao	196935a8d2	HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731 ). Contributed by tomscut. Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-12-02 19:10:35 +05:30
litao	1ebdac2cd6	HDFS-16361. Fix log format for QueryCommand (#3732 ). Contributed by tomscut.	2021-12-01 12:48:21 +05:30
litao	829af89dc4	HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-29 17:48:10 +09:00
litao	86705eda82	HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-26 19:52:14 +09:00
Takanobu Asanuma	9c887e5b82	HADOOP-18014. CallerContext should not include some characters. (#3698 ) Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Mingliang Liu <liuml07@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-25 14:05:04 +09:00
Viraj Jasani	cdc13e91b6	HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711 )	2021-11-25 10:23:04 +08:00
huhaiyang	99b161dec7	HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708 )	2021-11-25 10:20:42 +08:00
Viraj Jasani	e8566b3812	HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-24 22:54:11 +09:00
Mukul Kumar Singh	faa4eeacb1	HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694 )	2021-11-23 13:36:14 +00:00
litao	0ed817babb	HDFS-16337. Show start time of Datanode on Web (#3682 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-11-23 10:18:30 +08:00
Symious	c88640c4ad	HDFS-16320. Datanode retrieve slownode information from NameNode (#3654 )	2021-11-22 18:04:32 +08:00
litao	20b78c8f97	HDFS-16335. Fix HDFSCommands.md (#3681 )	2021-11-22 17:07:52 +09:00
Hui Fei	8e9ad2d7af	Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630 )" (#3697 )	2021-11-22 10:05:21 +08:00
GuoPhilipse	3d1ca6f6dc	HDFS-16334. Correct NameNode ACL description (#3678 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-22 07:41:57 +09:00
litao	965e3874e9	HDFS-16326. Simplify the code for DiskBalancer (#3663 ) Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-11-19 11:37:10 +09:00
GuoPhilipse	0c1eb0125b	HDFS-16328. Correct disk balancer desc (#3669 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-19 01:02:01 +09:00
Viraj Jasani	38614d05e1	HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-19 00:55:39 +09:00
litao	dc751df63b	HDFS-16329. Fix log format for BlockManager (#3670 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-19 00:42:01 +09:00
litao	5b05068fdc	HDFS-16310. RBF: Add client port to CallerContext for Router (#3635 )	2021-11-18 13:46:35 +09:00
huhaiyang	91af256a5b	HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630 )	2021-11-17 22:41:06 +08:00
Viraj Jasani	646c470e5d	HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659 )	2021-11-17 22:17:09 +09:00
Neil	5626734a36	HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320 )	2021-11-16 18:21:50 +08:00
litao	c9f95b01ef	HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643 ) Reviewed-by: Hui Fei <ferhui@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-11-16 11:19:14 +09:00
Symious	89fcbd84f9	HDFS-16313. RBF: Add metrics for each sub cluster (#3638 )	2021-11-16 09:47:25 +08:00
Viraj Jasani	e14a2dcbba	HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648 ) Reviewed-by: Ahmed Hussein <ahussein@apache.org>	2021-11-15 22:57:24 +09:00
GuoPhilipse	6e6f2e4baa	HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655 ). Contributed by guo. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-11-14 19:59:15 +05:30
huhaiyang	db89a9411e	HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596 )	2021-11-12 10:07:34 +08:00
jianghuazhu	3d692c0936	HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629 )	2021-11-12 09:52:02 +08:00
litao	1c1cf64616	HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-11-12 10:23:25 +09:00
litao	2a1a11c039	HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-10 19:22:32 +09:00
litao	ea65fc26d8	HDFS-16298. Improve error msg for BlockMissingException (#3615 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-10 19:16:45 +09:00
jianghuazhu	b307f18a58	HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622 )	2021-11-10 11:32:52 +09:00
litao	cbc7fb6bca	HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-11-09 17:06:28 +09:00
GuoPhilipse	9b2fa7e47a	HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability	2021-11-08 10:21:39 -08:00
Gautham B A	0a952e4d1f	HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625 )	2021-11-08 08:58:07 -08:00
Symious	d2b8d6b534	HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621 )	2021-11-08 09:52:16 +08:00
Xiangyi Zhu	e5cee76785	HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2021-11-08 09:30:53 +09:00
Symious	6f7b965808	HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613 )	2021-11-05 11:03:07 +08:00
Gautham B A	1032724aa3	HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617 )	2021-11-04 09:19:11 -07:00
jianghuazhu	72718a8dc1	HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605 )	2021-11-04 16:43:25 +09:00
jianghuazhu	18b2cca74c	HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602 )	2021-11-04 09:51:11 +08:00
litao	359b03c83e	HDFS-16266. Add remote port information to HDFS audit log (#3538 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-11-04 10:16:03 +09:00
daimin	a21895a5b3	HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593 )	2021-11-03 20:19:56 +00:00
Stephen O'Donnell	2f35cc36cd	HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598 )	2021-11-02 11:14:48 +00:00
jianghuazhu	618fea27d2	HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-11-02 00:55:50 +09:00
jianghuazhu	0a68413a4f	HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600 ) HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600). Contributed by JiangHua Zhu.	2021-10-31 08:48:52 +05:30
Masatake Iwasaki	634f8a2209	HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374 )	2021-10-31 09:29:34 +09:00
litao	6aa664c624	HDFS-16279. Print detail datanode info when process first storage report (#3564 )	2021-10-29 10:27:00 +09:00
litao	c1a8285363	HDFS-16281. Fix flaky unit tests failed due to timeout (#3574 )	2021-10-26 14:54:11 +08:00
Ahmed Hussein	62c86eaa0e	HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566 )	2021-10-23 16:13:07 -05:00
daimin	6e57449350	HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576 ). Contributed by daimin. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-10-22 21:12:13 +05:30
Ayush Saxena	f67e6c315b	Revert "Remove duplicate generic usage information to hdfs debug command (#3576 ). Contributed by daimin." Misses Jira ID in the commit Message: HDFS-16282 This reverts commit `7b913bc6bc`.	2021-10-22 21:10:38 +05:30
daimin	7b913bc6bc	Remove duplicate generic usage information to hdfs debug command (#3576 ). Contributed by daimin. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-10-22 21:09:37 +05:30
litao	2c37bebac4	HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568 ). Contributed by tomscut. Reviewed-by: Hui Fei <ferhui@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-10-22 08:21:30 +05:30
GuoPhilipse	1385cdfeb9	HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559 ). Contributed by guo. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-10-22 07:59:27 +05:30
Michael Kuchenbecker	a73ff6915a	HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571 )	2021-10-21 13:41:46 -07:00
Gautham B A	a2f089df4d	HDFS-16278. Make HDFS snapshot tools cross platform (#3563 )	2021-10-21 10:24:33 -07:00
Viraj Jasani	516f36c6f1	HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555 )	2021-10-21 16:54:25 +09:00
daimin	5337bebcc5	HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548 )	2021-10-18 11:55:55 +01:00
Xiangyi Zhu	d09b380dc2	HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557 ) Co-authored-by: zhuxiangyi <43412979@qq.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-10-18 13:53:41 +09:00
Ayush Saxena	263e4b595d	HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang.	2021-10-16 14:35:18 +05:30
litao	d72e2cb786	HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554 ) * HDFS-16274. Improve log for FSNamesystem#startFileInt * fix failed unit test	2021-10-15 22:41:38 +05:30
Viraj Jasani	1151edf12e	HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-10-14 13:07:24 +09:00
LeonGao	7279fe8661	HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546 )	2021-10-13 18:14:03 -07:00
Gautham B A	107fe227eb	HDFS-16267. Make hdfs_df tool cross platform (#3542 )	2021-10-13 09:11:21 -07:00
jianghuazhu	ff24fe6197	HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497 )	2021-10-13 17:21:03 +08:00
Gautham B A	a05eb23a85	HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532 )	2021-10-11 18:25:28 -07:00
jianghuazhu	d7c01d083c	HDFS-16264.When adding block keys, the records come from the specific Block Pool. (#3533 )	2021-10-11 19:49:58 +08:00
langlaile1221	88d8c3d0c8	HDFS-16241. Standby close reconstruction thread (#3493 ) Co-authored-by: zhanghuazong <Zhz@20068867>	2021-10-11 16:11:31 +08:00
Viraj Jasani	3ac0cc9e6f	HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530 ) Reviewed-by: Ahmed Hussein <ahussein@apache.org>	2021-10-11 15:32:49 +09:00
Gautham B A	4f3dfb7c1c	HDFS-16265. Refactor HDFS tool tests for better reuse (#3536 )	2021-10-09 11:00:43 -07:00
Gautham B A	cc95fc4cc6	HDFS-16263. Add CMakeLists for hdfs_allowSnapshot (#3531 )	2021-10-08 08:49:56 -07:00
Gautham B A	6bad3badf5	HDFS-16251. Make hdfs_cat tool cross platform (#3523 )	2021-10-07 10:57:11 -07:00
Viraj Jasani	8071dbb9c6	HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-10-07 10:58:29 +09:00
Gautham B A	e12cd0c638	HDFS-16254. Cleanup protobuf on exit of hdfs_allowSnapshot (#3518 )	2021-10-06 09:24:27 -07:00
S O'Donnell	a30231ff8e	HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell	2021-10-06 15:04:04 +01:00
Gautham B A	84f10fd78b	HDFS-16250. Refactor AllowSnapshotMock using GMock (#3513 )	2021-10-05 09:16:42 -07:00
huhaiyang	68c2accc20	HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511 )	2021-10-04 23:52:26 +08:00
Ayush Saxena	5f0452602f	HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422 ). Contributed by Ayush Saxena. Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>	2021-10-03 10:02:35 +05:30
Neil	bf9106c812	HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494 ) Contributed by Max Xie	2021-10-01 09:35:14 +02:00
jianghuazhu	1789c7c8c0	HDFS-16238.Improve comments related to EncryptionZoneManager. (#3483 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-10-01 10:30:41 +08:00
huhaiyang	1f8f53f7d7	HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471 )	2021-09-28 13:09:18 +08:00
jianghuazhu	02993bc11e	HDFS-16237.Record the BPServiceActor information that communicates with Standby. (#3479 )	2021-09-28 13:07:36 +08:00
Masatake Iwasaki	86e80752d1	HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. (#3492 )	2021-09-28 13:35:35 +09:00
Chao Sun	1397cf2496	Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" This reverts commit `59f01a548e`.	2021-09-25 21:32:43 -07:00
AngersZhuuuu	5f9321a5d4	HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472 )	2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang	87632bbacf	HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468 )	2021-09-24 08:35:23 -07:00
litao	3113a119af	HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456 )	2021-09-24 13:14:25 +08:00
Ayush Saxena	98e2781dde	HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov.	2021-09-23 10:45:02 +05:30
Gautham B A	35a8d48872	HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388 )	2021-09-22 10:29:38 -07:00
Viraj Jasani	c7e7b2f907	HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452 )	2021-09-22 15:34:51 +09:00
LeonGao	b1431813d0	HDFS-10648. Expose Balancer metrics through Metrics2 (#3427 )	2021-09-21 12:26:17 -07:00
Viraj Jasani	feee41aa00	HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386 ) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-09-21 10:11:20 +09:00
Tamas Domok	f93e8fbf2d	HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok * HDFS-16129. Fixing the signature secret file misusage in HttpFS. The signature secret file was not used in HttpFs. - if the configuration did not contain the deprecated httpfs.authentication.signature.secret.file option then it used the random secret provider - if both option (httpfs. and hadoop.http.) was set then the HttpFSAuthenticationFilter could not read the file because the file path was not substituted properly !NOTE! behavioral change: the deprecated httpfs. configuration values are overwritten with the hadoop.http. values. The commit also contains a follow up change to the YARN-10814, empty secret files will result in a random secret provider. Co-authored-by: Tamas Domok <tdomok@cloudera.com>	2021-09-20 14:29:50 +02:00
Thomas Leplus	f92c6750ec	HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455 ). Contributed by Thomas Leplus.	2021-09-20 00:09:05 +05:30
Viraj Jasani	c9763a99c1	HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429 )	2021-09-18 19:02:45 +08:00
jianghuazhu	cd5c6395e8	HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451 ). Contributed by JiangHua Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-18 11:06:51 +05:30
Viraj Jasani	7c25a77911	HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-09-17 16:39:29 +09:00
LeonGao	deb52cdcbb	HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425 )	2021-09-16 23:59:36 -07:00
lfengnan	4d21655d04	HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-09-16 14:58:02 +09:00
litao	2d479309cc	HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366 ) Reviewed-by: Hui Fei <ferhui@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-09-16 10:00:02 +09:00
wangzhaohui	3ecaa39668	HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317 ) Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>	2021-09-15 09:56:51 -07:00
litao	4f563ff1ba	HDFS-16225. Fix typo for FederationTestUtils (#3428 ). Contributed by tomscut.	2021-09-14 14:28:19 +05:30
jianghuazhu	1032104b9a	HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831 ). Contributed by JiangHua Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-14 14:24:36 +05:30
EungsopYoo	c4c5883d8b	HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359 ) Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-09-14 13:18:15 +08:00
Ayush Saxena	d9eb5ad6d3	HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404 ). Contributed by Ayush Saxena.	2021-09-13 22:50:39 +05:30
Ayush Saxena	17c58ac959	HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424 ). Contributed by Ayush Saxena.	2021-09-13 17:54:38 +08:00
bshashikant	356ebbbe80	HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340 )	2021-09-13 10:25:19 +05:30
jianghuazhu	edfde6eebc	HDFS-16197.Simplify getting NNStorage in FSNamesystem. (#3357 )	2021-09-12 19:58:30 +08:00
Symious	7f6553af75	HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421 )	2021-09-12 19:52:47 +08:00
LeonGao	90bc688c78	HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346 ) Contributed by Leon Gao * Router to support resolving monitored namenodes with DNS * Style * fix style and test failure * Add test for NNHAServiceTarget const * Resolve comments * Fix test * Comments and style * Create a simple function to extract port * Use LambdaTestUtils.intercept * fix javadoc * Trigger Build	2021-09-10 16:40:08 -07:00
Symious	99cb2b6b7f	HDFS-16065. RBF: Add metrics to record Router's operations (#3100 )	2021-09-09 18:45:23 +08:00
Ahmed Hussein	bddc9bf63c	HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375 )	2021-09-08 21:21:16 -07:00
Symious	c0890e6d04	HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379 )	2021-09-09 09:57:27 +08:00
litao	5a306955a4	HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378 )	2021-09-08 12:38:55 +08:00
jianghuazhu	4e4d4fcc7c	HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368 )	2021-09-06 18:06:12 +08:00
Renukaprasad C	dd6b2cebed	HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244 ). Contributed by Renukaprasad C.	2021-09-05 17:07:59 +08:00
litao	364a1fbfb3	HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354 ). Contributed by tomscut.	2021-09-05 14:19:01 +08:00
Viraj Jasani	cc6b545365	HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-04 16:20:08 +05:30
Weison Wei	a610f6d9c3	HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367 ). Contributed by Weison Wei. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-09-03 12:05:53 +05:30
langlaile1221	24af52efab	HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894 )	2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G	164608b546	HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353 ) Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2021-08-31 12:25:03 +08:00
Viraj Jasani	aa9cdf2af6	HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-26 00:37:38 -07:00
LeonGao	b53cae0ffb	HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284 ) * Add DNS resolution for QJM * Add log * Resolve comments * checkstyle * typo	2021-08-25 17:40:12 -07:00
Viraj Jasani	1b9927afe1	HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-08-25 17:37:25 +09:00
Neil	9084c728eb	HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315 )	2021-08-24 11:15:47 +08:00
Gautham B A	b6d1971820	HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311 )	2021-08-20 10:04:55 -07:00
litao	07627ef19e	HDFS-16177. Bug fix for Util#receiveFile (#3310 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-08-19 12:30:38 +08:00
litao	0e922aab41	HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291 )	2021-08-19 09:51:00 +08:00
Gautham B A	409878ef80	HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303 )	2021-08-17 09:49:52 -07:00
jianghuazhu	53a2c65694	HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-17 16:47:10 +08:00
Viraj Jasani	6342d5e523	HDFS-16171. De-flake testDecommissionStatus (#3280 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-16 14:54:25 +09:00
Viraj Jasani	919a99bbb4	HDFS-16163. Avoid locking entire blockPinningFailures map (#3296 )	2021-08-16 13:40:05 +08:00
Gautham B A	77383a4d55	HDFS-15976. Make mkdtemp cross platform (#2908 )	2021-08-10 09:23:28 -07:00
Akira Ajisaka	4972e7a246	Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk.	2021-08-07 11:27:25 +09:00
wangzhaohui	e85c44657c	HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270 ) Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-08-06 18:23:00 +08:00
wangzhaohui	a73b64f86b	HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269 ). Contributed by wangzhaohui.	2021-08-06 09:27:05 +01:00
zhangshuyan0	10a2526b0b	HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247 ) Contributed by Shuyan Zhang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-08-04 00:22:21 +08:00
Viraj Jasani	ccfa072dc7	HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang	6d77f3b6cd	HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-30 08:29:56 -07:00
jianghuazhu	6f730fd25c	HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local> Reviewed-by: cxorm <lianp964@gmail.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-07-28 21:57:28 -07:00
He Xiaoqiao	683feaa1d4	HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>	2021-07-29 11:19:17 +08:00
S O'Donnell	4eae284827	HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell	2021-07-28 14:49:23 +01:00
zhihaixu2012	ae20516ebc	HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175 ) Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a Co-authored-by: Zhihai Xu <zxu@apache.org>	2021-07-27 19:18:44 -07:00
JiaguodongF	aecfcf165f	HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185 ) * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid * HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-27 22:25:21 +05:30
Viraj Jasani	b038042ece	HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228 ). Contributed by Viraj Jasani. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-27 14:57:12 +08:00
Akira Ajisaka	d710ec8d85	HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-07-26 16:46:07 +09:00
litao	5d765497c5	HDFS-16131. Show storage type for failed volumes on namenode web (#3211 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-07-26 12:45:01 +08:00
Akira Ajisaka	2f2f822488	HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227 ) Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin" This reverts commit `b6d839a60c`. Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml	2021-07-26 01:53:39 +09:00
Eric Yin	de41ce8a16	HDFS-16087. Fix stuck issue in rbfbalance tool (#3141 ). Contributed by Eric Yin.	2021-07-21 00:01:55 +08:00
He Xiaoqiao	6ed7670a93	HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C.	2021-07-17 21:57:00 +08:00
Kihwal Lee	47002719f2	HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee.	2021-07-16 14:22:39 -05:00
crossfire	4bb25c810b	HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414 )	2021-07-15 19:58:12 -07:00
Abhishek Das	1dd03cc4b5	HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260 )	2021-07-13 18:11:50 -07:00
LeonGao	ea90c5117d	HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639 ) * Rebase trunk * Fix to use FQDN and update config name * Fix javac * Style and trigger build * Trigger Build after force push * Trigger Build * Fix config names	2021-07-13 14:18:59 -07:00
Artem Smotrakov	ba325a8ada	HADOOP-17793. Better token validation (#3189 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-10 13:42:31 +09:00
S O'Donnell	b6c06c4b76	HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma	2021-07-09 10:03:57 +01:00
bshashikant	6450c1ea9a	HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188 )	2021-07-09 10:26:42 +05:30
Viraj Jasani	618c9218ee	HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171 ) Reviewed-by: Steve Loughran <stevel@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-07-08 16:03:40 +09:00
litao	b4c2647d0d	HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140 )	2021-07-08 14:10:45 +08:00
JiaguodongF	9381071758	HDFS-16114. the balancer parameters print error (#3179 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-07-07 14:13:05 +05:30
Ayush Saxena	c85a18327d	HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w.	2021-07-06 23:26:52 +05:30
litao	a447cfd78b	HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-07-05 19:31:22 -07:00
Viraj Jasani	da575a53cf	HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169 )	2021-07-05 09:23:24 +08:00
litao	b87bac13e4	HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-07-05 08:14:37 +09:00
litao	390f8603d3	HDFS-16106. Fix flaky unit test TestDFSShell (#3168 )	2021-07-02 15:31:25 +08:00
zhangshuyan0	9be17339eb	HDFS-16099. Make bpServiceToActive to be volatile. (#3159 ) Co-authored-by: zhangshuyan <zqingchai@gmail.com>	2021-07-01 23:52:21 +08:00
litao	52b9319e57	HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160 ). Contributed by tomscut. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-07-01 18:44:10 +05:30
litao	56c7ada7a5	HDFS-16086. Add volume information to datanode log for tracing (#3136 )	2021-06-30 22:06:56 -07:00
Viraj Jasani	d2e7003400	HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148 ) Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: stack Chuang <stack@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-30 12:55:07 -07:00
Xiangyi Zhu	4cac6ec405	HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156 ) Co-authored-by: zhuxiangyi <43412979@qq.com> Reviewed-by: Viraj Jasani <vjasani@apache.org>	2021-06-30 00:12:08 -07:00
litao	95454d821c	HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146 )	2021-06-29 03:15:12 -07:00
Viraj Jasani	93a1685073	HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150 )	2021-06-29 02:31:02 -07:00
He Xiaoqiao	748570b73c	Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." This reverts commit `0d07837712`.	2021-06-25 22:38:34 +08:00
He Xiaoqiao	0d07837712	HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun.	2021-06-25 01:09:42 +08:00
litao	f78b6d47f9	HDFS-16085. Move the getPermissionChecker out of the read lock (#3134 ). Contributed by tomscut. Signed-off-by: Takanobu Asanuma <tasanuma@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-24 15:59:42 +05:30
litao	fdef2b4cca	HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117 )	2021-06-24 11:27:40 +09:00
Gautham B A	ef5dbc7283	HADOOP-17766. CI for Debian 10 (#3129 )	2021-06-23 10:02:33 -07:00
Viraj Jasani	d9fbb3c508	HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127 )	2021-06-22 20:21:34 -07:00
He Xiaoqiao	10b79a26fe	HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. Co-authored-by: Tao Yang <taoyang1@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-06-22 00:32:55 +08:00
Viraj Jasani	c488abbc79	HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115 ) Reviewed-by: Hui Fei <ferhui@apache.org>	2021-06-21 10:25:12 +09:00
litao	bdc9c8809e	HDFS-16079. Improve the block state change log (#3120 ). Contributed by tomscut. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 17:23:11 +05:30
Viraj Jasani	d9eccb1eb3	HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 17:19:05 +05:30
Ahmed Hussein	c255feb5d7	HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095 ). Contributed by Ahmed Hussein. Reviewed-by: Jim Brennan <jbrennan@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:22:47 +05:30
litao	c491f81a30	HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119 ). Contributed by tomscut. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-06-20 10:20:52 +05:30
LeonGao	643dfd60e4	HDFS-15842. HDFS mover to emit metrics. (#2738 )	2021-06-19 15:39:46 -07:00
Wei-Chiu Chuang	1ad674e548	HDFS-16074. Remove an expensive debug string concatenation (#3107 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org> Reviewed-by: Viraj Jasani <vjasani@apache.org> Reviewed-by: litao <tomleescut@gmail.com>	2021-06-16 20:37:56 -07:00
hust_hhb	56d249759f	HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065 )	2021-06-17 11:08:29 +08:00
Ayush Saxena	2b304ad645	HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w.	2021-06-16 15:53:12 +05:30
Wei-Chiu Chuang	a77bf7cf07	Make upstream aware of 3.3.1 release	2021-06-15 15:43:36 +00:00
Takanobu Asanuma	839fcf7682	HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104 )	2021-06-15 13:20:09 +05:30
Viraj Jasani	c1bf3cb0da	HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998 ) Contributed by Viraj Jasani	2021-06-14 14:23:50 -07:00
Siyao Meng	ebee2aed00	HDFS-16055. Quota is not preserved in snapshot INode (#3078 )	2021-06-14 10:48:22 -07:00
Ahmed Hussein	352949d070	HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097 )	2021-06-11 11:00:42 -05:00
litao	51ebf7b2a0	HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084 ) Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-06-11 17:30:55 +09:00
Viraj Jasani	4ef27a596f	HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087 )	2021-06-11 12:15:52 +09:00
Viraj Jasani	eefa664fea	HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073 )	2021-06-09 15:12:48 +09:00
Ayush Saxena	a1a3184171	HDFS-15916. Addendum. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#3056 )	2021-06-08 19:34:45 -07:00
Ahmed Hussein	a2a0283c7b	HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058 )	2021-06-08 11:09:31 -05:00
litao	c748fce17a	HDFS-16048. RBF: Print network topology on the router web (#3062 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2021-06-08 16:14:06 +09:00
huhaiyang	b38b00e528	HDFS-15998. Fix NullPointException In listOpenFiles (#3036 ) Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>	2021-06-01 00:26:47 -07:00
Ayush Saxena	b8ab19373d	Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 )" This reverts commit `041488e8f5`.	2021-06-01 08:38:13 +05:30
Ayush Saxena	bce14e746b	HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng.	2021-06-01 08:34:56 +05:30
Jinglun	e90c41af34	HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-30 19:28:40 +08:00
Jinglun	2b11e160e2	Revert "HDFS-15973." because the user.name and user.email message is not correct. This reverts commit `b7f69467c1`.	2021-05-30 19:27:10 +08:00
李经纶	b7f69467c1	HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-30 18:50:03 +08:00
Konstantin V Shvachko	8ce30f51f9	HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.	2021-05-27 18:47:20 -07:00
Konstantin V Shvachko	1abd03d68f	HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.	2021-05-26 14:59:50 -07:00
Gautham B A	bcaeb1ac8c	Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898 )"" (#3044 )	2021-05-26 14:12:53 -07:00
Hui Fei	98a74e2351	HDFS-16041. TestErasureCodingCLI fails (#3052 )	2021-05-26 20:12:17 +09:00
Viraj Jasani	986d0a4f1d	HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-26 17:14:31 +09:00
Narges Shadab	3fdeb7435a	HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027 )	2021-05-26 14:43:41 +09:00
zhuxiangyi	e18d806212	HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009 )	2021-05-26 13:41:49 +08:00
Viraj Jasani	028ec4704b	HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-05-25 10:55:58 +09:00
lfengnan	43bf009112	HDFS-15757 RBF: Improving Router Connection Management (#2651 )	2021-05-19 10:53:42 -07:00
huhaiyang	acd712ca92	HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747 )	2021-05-19 09:04:41 +08:00
Ayush Saxena	fcd4140e5f	HDFS-15988. Stabilise HDFS Pre-Commit. (#2860 ). Contributed by Ayush Saxena. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-05-16 00:45:11 +05:30
Viraj Jasani	b944084b32	HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-11 12:38:15 +09:00
Siyao Meng	8d5cc98b42	HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958 )	2021-05-10 12:41:07 -07:00
Ayush Saxena	359c0c1d1e	HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w.	2021-05-10 17:09:51 +05:30
Jinglun	1b6994287a	HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819 ). Contributed by zhuobin zheng. Reviewed-by: Jinglun <jinglun@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-05-08 19:50:17 +08:00
Akira Ajisaka	9aa651f0b4	HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980 ) Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>	2021-05-08 00:24:06 +09:00
zhuqi	76c69c39d9	HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833 ) Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-05-05 21:02:13 +05:30
Viraj Jasani	041488e8f5	HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927 ) Reviewed-by: Siyao Meng <siyao@apache.org>	2021-05-04 01:31:23 -07:00
Akira Ajisaka	7c561e2a02	HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970 )	2021-05-04 00:25:29 +09:00
lfengnan	6e525ab81c	HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang	f1e1809029	HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945 ) Reviewed-by: Steve Loughran <stevel@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-05-02 11:10:32 +09:00
Karthik Palanisamy	bd3da73a0f	HDFS-15865. Interrupt DataStreamer thread if no ack (#2728 )	2021-05-01 23:35:31 +05:30
lfengnan	552e9dcc6c	HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954 ). Contributed by Fengnan Li. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang	b2e54762a4	HDFS-15624. fix the function of setting quota by storage type (#2377 ) (#2955 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org> Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44	2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang	f41a368c14	HADOOP-11245. Update NFS gateway to use Netty4 (#2832 ) Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>	2021-04-28 23:43:11 -07:00
bshashikant	ef13f8ad6b	HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881 )	2021-04-27 09:15:06 +05:30
Stephen O'Donnell	605ed85c29	HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849 ). Contributed by Stephen O'Donnell	2021-04-26 11:00:23 +01:00
litao	b968fa0957	HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-26 18:38:43 +09:00
Takanobu Asanuma	01bad0e92a	HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel.	2021-04-26 13:29:28 +09:00
Brahma Reddy Battula	2621d3f15b	HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang	90c6caf650	Revert "HDFS-15624. fix the function of setting quota by storage type (#2377 )" This reverts commit `394b9f7a5c`. Ref: HDFS-15995. Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3) Will re-work this fix again later.	2021-04-26 11:27:15 +08:00
jianghuazhu	c486dc396c	HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913 ) Contributed by JiangHua Zhu.	2021-04-25 14:31:51 +08:00
zhuxiangyi	dff95c5eca	HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915 ) Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-23 10:19:11 +09:00
Viraj Jasani	acb49f2fc3	HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-21 19:45:49 +09:00
litao	85a3532849	HDFS-15970. Print network topology on the web (#2896 )	2021-04-19 20:42:30 +09:00
litao	af0448d37b	HDFS-15975. Use LongAdder instead of AtomicLong (#2907 )	2021-04-17 16:49:09 +09:00
Eric Badger	aed13f0f42	Revert "HDFS-15971. Make mkstemp cross platform (#2898 )" This reverts commit `b088f46b68`.	2021-04-16 17:15:33 +00:00
zhangshuyan0	14816be0b1	HDFS-15963. Unreleased volume references cause an infinite loop. (#2889 ) Contributed by Shuyan Zhang. Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-17 00:08:31 +08:00
Akira Ajisaka	f0241ec216	HDFS-15977. Call explicit_bzero only if it is available. (#2914 ) Reviewed-by: Masatake Iwasaki <iwasakims@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2021-04-16 13:26:45 +09:00
lfengnan	17be99f9f9	HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903 )	2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian	c821008836	HDFS-15850. Superuser actions should be reported to external enforcers (#2784 )	2021-04-15 14:45:51 -07:00
Masatake Iwasaki	2717203f85	HADOOP-17569. Building native code fails on Fedora 33. (#2886 )	2021-04-15 21:33:18 +09:00
Ayush Saxena	7d6f0ca0bd	Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754 ). Contributed by tomscut." This reverts commit `e565b05c80`. The removed method needs to be used in HDFS-15423.	2021-04-13 23:28:42 +05:30
Gautham B A	b088f46b68	HDFS-15971. Make mkstemp cross platform (#2898 )	2021-04-13 08:54:45 -07:00
Inigo Goiri	82462739f8	Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605 )" (#2900 ) This reverts commit `cb3ed32fe0`.	2021-04-12 20:44:31 -07:00
lfengnan	cb3ed32fe0	HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605 )	2021-04-12 12:42:33 -07:00
Gautham B A	3148791da4	HDFS-15962. Make strcasecmp cross platform (#2883 )	2021-04-09 09:01:33 -07:00
Viraj Jasani	56bd968fb4	HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874 )	2021-04-09 11:10:18 +09:00
Gautham B A	bf66116407	HDFS-15955. Make explicit_bzero cross platform (#2875 )	2021-04-08 09:44:47 -07:00
Ayush Saxena	c6539e3289	HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863 ). Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2021-04-08 20:49:08 +05:30
Stephen O'Donnell	4c567fcff7	HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838 )	2021-04-08 11:59:02 +01:00
Gautham B A	42ddb5c6fe	HDFS-15909. Make fnmatch cross platform (#2792 )	2021-04-06 09:23:21 -07:00
He Xiaoqiao	459df41997	HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.	2021-04-06 21:22:48 +08:00
Gautham B A	36014b8282	HDFS-15949. Fix integer overflow (#2857 ) Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-04-06 18:22:54 +09:00
Viraj Jasani	3f2682b92b	HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 13:39:10 +09:00
Viraj Jasani	de05cefbae	HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844 ). Contributed by Viraj Jasani Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-06 00:19:21 +05:30
Gautham B A	6e1df1c048	HDFS-15947. Replace deprecated protobuf APIs (#2856 )	2021-04-05 09:52:48 -07:00
litao	9e2d5d6d05	HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-05 17:47:10 +09:00
Gautham B A	ca5e13fd74	HDFS-15950. Remove unused hdfs.proto import (#2858 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-04-05 11:42:05 +09:00
zhuqi	96e410a127	HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829 ). Contributed by Qi Zhu. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-04-04 03:11:02 +05:30
litao	3cb76447f5	HDFS-15946. Fix java doc in FSPermissionChecker (#2855 ). Contributed by tomscut. Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-02 10:37:04 -07:00
Gautham B A	70536ba1f9	HDFS-15944. Prevent truncation by snprintf (#2853 )	2021-04-02 09:37:33 -07:00
Brahma Reddy Battula	ed74479ea5	HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree.	2021-04-02 09:47:20 +05:30
litao	7c83f140dc	HDFS-15938. Fix java doc in FSEditLog (#2837 ). Contributed by tomscut. Signed-off-by: Akira Ajisaka <aajisaka@apache.org> Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 19:28:17 -07:00
litao	4bd04126d6	HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2021-04-02 10:33:40 +09:00
Viraj Jasani	4f28738010	HDFS-15931 : Fix non-static inner classes for better memory management (#2830 ). Contributed by Viraj Jasani Signed-off-by: Mingliang Liu <liuml07@apache.org>	2021-04-01 16:34:31 -07:00
Brahma Reddy Battula	0665ce9930	HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.	2021-04-01 09:19:39 +05:30
He Xiaoqiao	5eca748970	HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.	2021-04-01 11:32:03 +08:00
Borislav Iordanov	2c482fbacf	HADOOP-16524. Automatic keystore reloading for HttpServer2 Reapply of issue reverted first because it caused yarn failures and then again because the commit message was incorrectly formatted (and yet again because of commit message format). Signed-off-by: stack <stack@apache.org>	2021-03-31 10:46:35 -07:00
stack	22961a615d	Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" This reverts commit `a2975d2153`.	2021-03-31 10:43:09 -07:00
stack	a2975d2153	HADOOP-16524. Automatic keystore reloading for HttpServer2 Reapply of issue reverted first because it caused yarn failures and then again because the commit message was incorrectly formatted.	2021-03-31 10:40:20 -07:00
stack	5183aaeda2	Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693 )" Revert to fix the summary message. This reverts commit `9509bebf7f`.	2021-03-31 10:39:55 -07:00
Borislav Iordanov	9509bebf7f	Hadoop 16524 - resubmission following some unit test fixes (#2693 ) Signed-off-by: stack <stack@apache.org>	2021-03-31 10:07:42 -07:00
He Xiaoqiao	da4ceba4aa	HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C.	2021-03-31 13:00:13 +08:00
Gautham B A	7dfff496fa	HDFS-15929. Replace RAND_pseudo_bytes in util.cc (#2826 )	2021-03-30 13:36:02 -07:00
Gautham B A	8668abf87e	HDFS-15927. Catch polymorphic type by reference (#2824 )	2021-03-30 13:35:45 -07:00
Gautham B A	50b1e1c0c4	HDFS-15928. Replace RAND_pseudo_bytes in rpc_engine.cc (#2825 )	2021-03-30 10:37:10 -07:00
Viraj Jasani	90fdd04f54	HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823 ). Contributed by Viraj Jasani. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-29 23:22:54 +05:30
Gautham B A	6a3e135316	HDFS-15922. Use memcpy for copying non-null terminated string. (#2827 )	2021-03-29 10:38:59 -07:00
Harunobu Daikoku	ea6595d3b6	HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787 ) Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>	2021-03-29 11:43:29 +09:00
Ayush Saxena	cd2501e54b	HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.	2021-03-28 16:13:59 +05:30
litao	72037a63b1	HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748 ) Reviewed-by: Dinesh Chitlangia <dineshc@apache.org> Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2021-03-27 17:57:09 +09:00
Gautham B A	1fed18bb2d	HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c (#2818 )	2021-03-26 09:26:35 -07:00
Akira Ajisaka	af1f9f43ea	HADOOP-17133. Implement HttpServer2 metrics (#2145 )	2021-03-25 12:09:43 -07:00
He Xiaoqiao	ed5814deff	HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.	2021-03-25 12:24:14 +08:00
Gautham B A	654555783d	HDFS-15918. Replace deprecated RAND_pseudo_bytes (#2811 )	2021-03-24 10:52:33 -07:00
touchida	95e6892675	HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585 )	2021-03-24 16:56:09 +08:00
Takanobu Asanuma	b4d97a8dc7	HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel.	2021-03-24 17:51:35 +09:00
Viraj Jasani	4b4ccce02f	HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 22:47:45 -07:00
Ayush Saxena	569e407f64	HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.	2021-03-24 02:10:02 +05:30
Gautham B A	714427c414	HDFS-15910. Improve security with explicit_bzero (#2793 )	2021-03-23 09:58:19 -07:00
Jack Jiang	9c2a712597	HDFS-15913. Remove useless NULL checks before instanceof (#2805 )	2021-03-23 08:51:35 -07:00
Narges Shadab	d77c7ab4e0	HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790 ). Contributed by Narges Shadab. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-23 17:06:58 +05:30
Gautham B A	61bfa42389	HDFS-15903. Refactor X-Platform lib (#2783 )	2021-03-22 08:40:54 -07:00
litao	343ce8a8c9	HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788 )	2021-03-22 20:29:17 +09:00
He Xiaoqiao	03a3a04c06	HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-22 18:37:00 +08:00
Viraj Jasani	261191cbc0	HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785 ) Contributed by Viraj Jasani. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2021-03-19 20:37:44 -07:00
Narges Shadab	f5f3fc6fe9	HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736 ). Contributed by Narges Shadab. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-19 16:36:48 +08:00
Viraj Jasani	9c43b60348	HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772 )	2021-03-17 15:12:36 +09:00
Takanobu Asanuma	9ba60c33d8	HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.	2021-03-17 14:14:26 +09:00
touchida	25390babc7	HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726 ) Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-03-17 12:00:05 +09:00
Hui Fei	0d42104119	HDFS-15898. Test case TestOfflineImageViewer fails (#2779 )	2021-03-17 09:11:30 +08:00
Gautham B A	47620f8821	HDFS-15843. Make write cross-platform (#2710 )	2021-03-16 09:06:51 -07:00
sunlisheng	7025f39944	HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.	2021-03-15 11:34:13 +08:00
He Xiaoqiao	970455c917	HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.	2021-03-14 17:48:18 +08:00
litao	e565b05c80	HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754 ). Contributed by tomscut.	2021-03-14 04:07:31 +05:30
lamberken	6921ec8b40	HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751 ). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-13 14:22:11 +08:00
Xiaoyu Yao	7cf5969f0b	HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764 )	2021-03-12 11:27:53 -08:00
Takanobu Asanuma	ecd3335187	HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.	2021-03-12 15:42:15 +09:00
Akira Ajisaka	23b343aed1	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753 ) Removed findbugs from the hadoop build images and added spotbugs instead. Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>	2021-03-11 10:56:07 +09:00
Hui Fei	6a55baeee4	HDFS-15875. Check whether file is being truncated before truncate (#2746 )	2021-03-10 14:11:29 +08:00
litao	6a922f9896	HDFS-15873. Add namenode address in logs for block report (#2743 )	2021-03-08 11:50:51 -05:00
He Xiaoqiao	4a0b7f7ebe	HDFS-15384. Document getLocatedBlocks(String src, long start) of DFSClient only return partial blocks. Contributed by Yang Yun.	2021-03-08 12:18:46 +08:00
tomscut	9cb51bf106	HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668 ) Contributed by tomscut.	2021-03-06 13:40:44 -08:00
litao	cdba06e380	HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739 ) Co-authored-by: tomscut <litao@bigo.sg>	2021-03-03 22:34:02 +09:00
JJiaguodong	8af56de1fa	HADOOP-17560. Fix some spelling errors (#2730 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-03-03 11:41:05 +09:00
zhuqi	1f1a1ef52d	HDFS-15856: Make write pipeline retry times configurable. (#2721 ). Contributed by Qi Zhu Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-02 13:16:11 +08:00
Konstantin V Shvachko	9501c698f4	HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.	2021-03-01 17:17:23 -08:00
litao	32353eb38a	HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718 ) Authored-by: tomscut <litao@bigo.sg>	2021-03-01 07:52:59 -08:00
S O'Donnell	c3b3b36dee	HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell	2021-03-01 11:36:41 +00:00
S O'Donnell	9c583f36d5	HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee	2021-02-24 10:29:36 +00:00
LeonGao	940c780feb	HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704 )	2021-02-23 22:29:48 -08:00
Ayush Saxena	647d23b611	HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.	2021-02-23 19:08:38 +05:30
Takanobu Asanuma	a57282540f	HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714 )	2021-02-23 15:12:06 +09:00
Wang Yuxuan	1e32ab7d2f	HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang	f8059d0bdf	Revert "first commit (#2559 )" This reverts commit `1b94569d89`.	2021-02-22 18:13:06 +08:00
Wang Yuxuan	1b94569d89	first commit (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 02:10:39 -08:00
jianghuazhu	98ffd65d8b	HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>	2021-02-21 23:22:32 -08:00
Ayush Saxena	ff84a57483	HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.	2021-02-20 19:48:00 +05:30
Ayush Saxena	2ce5752fa8	HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun.	2021-02-20 19:34:29 +05:30
Takanobu Asanuma	25af8901a9	HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.	2021-02-19 14:44:58 +09:00
Akira Ajisaka	9a298d180d	Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 )" This reverts commit `4cf3531583`.	2021-02-19 11:09:10 +09:00
Hui Fei	2970bd93f3	HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694 )	2021-02-19 09:07:22 +08:00
Ayush Saxena	e391844e8e	HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.	2021-02-18 01:39:04 +05:30
Akira Ajisaka	0ed9e36282	HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696 )	2021-02-17 01:59:24 -05:00
Akira Ajisaka	4cf3531583	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 ) Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images, and added spotbugs in the images instead. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>	2021-02-17 10:38:20 +09:00
Akira Ajisaka	2b3c5b1733	HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-02-16 14:50:50 +09:00
Zehao Chen	07a4220cd2	HDFS-15821. Add metrics for in-service datanodes (#2690 ). Contributed by Zehao Chen.	2021-02-15 11:14:32 -06:00
litao	c3134ab3a9	HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676 )	2021-02-14 00:12:24 -05:00
Steve Loughran	798df6d699	HADOOP-13327 Output Stream Specification. (#2587 ) This defines what output streams and especially those which implement Syncable are meant to do, and documents where implementations (HDFS; S3) don't. With tests. The file:// FileSystem now supports Syncable if an application calls FileSystem.setWriteChecksum(false) before creating a file -checksumming and Syncable.hsync() are incompatible. Contributed by Steve Loughran.	2021-02-10 10:28:59 +00:00
LeonGao	9434c1eccc	HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679 )	2021-02-09 10:49:28 +00:00
bshashikant	2df2dfb9ed	HDFS-15817. Rename snapshots while marking them deleted. (#2677 )	2021-02-09 10:27:51 +05:30
LeonGao	0e2b3086e3	HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625 )	2021-02-08 11:01:21 -08:00
bshashikant	19ae0faacc	HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682 )	2021-02-06 08:26:12 -08:00
Jim Brennan	c4918fb298	[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan	2021-02-05 21:20:30 +00:00
fermi	7ef2875c89	HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang	2021-02-05 11:01:49 +08:00
maobaolong	b99dba38b9	HDFS-15819. Fix import style #2681	2021-02-05 10:11:27 +08:00
Hui Fei	15a1f7adfc	HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang	2021-02-04 09:19:51 +08:00
Kihwal Lee	182623d2bc	HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.	2021-02-03 08:55:50 -06:00
Hui Fei	66ecee333e	HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang	2021-02-03 20:00:45 +08:00
YaYun-Wang	394b9f7a5c	HDFS-15624. fix the function of setting quota by storage type (#2377 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>	2021-02-02 22:44:34 -08:00
crossfire	18978f2e20	HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657 )	2021-02-02 09:02:09 +00:00
He Xiaoqiao	1b893e10ae	HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-02 13:50:54 +08:00
belugabehr	21a3fc3d2d	HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633 )	2021-02-01 09:40:01 -08:00
He Xiaoqiao	b54134661b	HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-01 23:28:29 +08:00
Narges Shadab	115623a6ee	HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652 )	2021-02-01 01:56:59 -08:00
Siyao Meng	1a205cc3ad	HADOOP-17424. Replace HTrace with No-Op tracer (#2645 )	2021-02-01 13:42:44 +09:00
Gautham B A	ad483fd66e	HDFS-15740. Add x-platform utilities (#2567 )	2021-01-29 10:11:38 -08:00
Jim Brennan	f565476939	[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.	2021-01-28 18:17:15 +00:00
Jim Brennan	6184ab07f6	[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.	2021-01-28 16:47:52 +00:00
sunlisheng	f8769e0f4b	HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.	2021-01-28 10:10:39 +08:00
Ayush Saxena	e40f99f6d5	HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun.	2021-01-26 14:25:27 +05:30
Akira Ajisaka	45b3a84709	HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626 )	2021-01-21 10:07:46 +09:00
He Xiaoqiao	97f843de3a	HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.	2021-01-16 14:06:56 +08:00
LeonGao	7743d40ac5	HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583 ). Contributed by Leon Gao.	2021-01-15 16:28:11 -08:00
Masatake Iwasaki	630f8ddd2c	HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-14 16:19:35 +09:00
touchida	8ec824f2ba	HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598 ) Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-13 11:23:07 +09:00
Yiqun Lin	0d7ac54510	HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li.	2021-01-12 23:38:55 +08:00
Masatake Iwasaki	32abc0af49	HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507 )	2021-01-11 11:59:32 -08:00
He Xiaoqiao	ec22850dbe	Make upstream aware of 3.2.2 release.	2021-01-09 18:06:08 +08:00
Ahmed Hussein	77435a025e	HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601 ). Contributed by Ahmed Hussein and Daryn Sharp.	2021-01-08 13:10:09 -06:00
Ayush Saxena	2ba7ec2b48	HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun.	2021-01-08 16:27:54 +05:30
lfengnan	87bd4d2aca	HDFS-15754. Add DataNode packet metrics (#2578 ) Contributed by Fengnan Li.	2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang	2b4febcf57	HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533 )	2021-01-04 20:54:44 -08:00
Ayush Saxena	77299ae992	HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun.	2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang	66ee0a6df0	HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453 )	2021-01-04 09:43:58 -08:00
Masatake Iwasaki	2825d060cf	HDFS-15648. TestFileChecksum should be parameterized. (#2501 )	2021-01-04 14:00:57 +09:00
Ayush Saxena	1cd96e8dd8	HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.	2021-01-01 11:37:52 +05:30
maobaolong	3b77cf4446	HDFS-15749. Make size of editPendingQ can be configurable (#2572 )	2020-12-28 14:18:18 +08:00
hemanthboyina	16a20503ca	HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.	2020-12-22 17:27:31 +05:30
langlaile1221	4ae561bcb4	HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566 )	2020-12-22 09:25:11 +08:00
Ayush Saxena	ba496f61f8	HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.	2020-12-22 02:16:00 +05:30
Ayush Saxena	5ab1db9e21	HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut.	2020-12-21 09:58:30 +05:30
Ayush Saxena	2aea43bf4f	HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.	2020-12-20 23:06:19 +05:30
Ayush Saxena	3ecd3628d4	HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.	2020-12-20 15:18:25 +05:30
Ayush Saxena	70789fc6b9	HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.	2020-12-20 15:12:32 +05:30
Ahmed Hussein	c2672bb234	HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-17 11:33:29 -06:00
Ayush Saxena	0da00fc743	HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-12-14 20:42:23 -08:00
liuyanpunk	3363cc380b	HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534 )	2020-12-15 10:06:11 +05:30
S O'Donnell	9ed737001c	HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell	2020-12-11 20:53:10 +00:00
Ahmed Hussein	be35fa186c	HDFS-15717. Improve fsck logging. (#2529 ) Contributed by Kihwal Lee and Ahmed Hussein	2020-12-11 11:02:42 -06:00
Neil	9bd3c9bc50	HDFS-15720 namenode audit async logger should add some log4j config (#2532 )	2020-12-10 13:47:00 -08:00
Ahmed Hussein	3ec01b1bb3	HDFS-15711. Add Metrics to HttpFS Server. (#2521 ) Contributed by Ahmed Hussein and Kihwal Lee	2020-12-10 13:09:53 -06:00
Ahmed Hussein	01383a2172	HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528 )	2020-12-08 13:25:24 -06:00
Gautham B A	4ffec79b7c	HDFS-15712. Upgrade googletest to 1.10.0 (#2523 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-12-08 17:08:14 +09:00
crossfire	40f7543a6d	HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518 )	2020-12-07 15:49:45 -08:00
Ahmed Hussein	32099e36dd	HDFS-15707. NNTop counts don't add up as expected. (#2516 )	2020-12-07 14:22:06 -06:00
Yiqun Lin	da1ea2530f	HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.	2020-12-07 18:52:12 +08:00
Ayush Saxena	ad40715690	HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.	2020-12-07 15:00:19 +05:30
Ayush Saxena	7dda804a1a	HDFS-14090. RBF: Improved isolation for downstream name nodes. {Static}. Contributed by Fengnan Li and CR Hota.	2020-12-04 22:56:37 +05:30
Hui Fei	e2c1268ebd	HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.	2020-12-04 09:20:09 +08:00
Ahmed Hussein	07655a7040	HDFS-15706. HttpFS: Log more information on request failures. (#2515 )	2020-12-03 15:14:40 -08:00
Wei-Chiu Chuang	42a29199c0	HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.	2020-12-02 21:32:02 -08:00
LeonGao	6ff2409b31	HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483 ). Contributed by Leon Gao.	2020-12-02 15:53:09 -08:00
Ahmed Hussein	60201cbf69	HDFS-15703. Don't generate edits for set operations that are no-op (#2508 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 15:38:20 -06:00
Ahmed Hussein	2b5b556dd7	HDFS-15695. NN should not let the balancer run in safemode (#2489 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 13:59:00 -06:00
Ahmed Hussein	918ba9e82f	HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487 ) Contributed by Kuhu Shukla and Ahmed Hussein	2020-12-01 15:02:53 -06:00
Masatake Iwasaki	6a1d7d9ed2	HDFS-15677. TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk. (#2503 )	2020-12-01 08:06:52 +09:00
Hui Fei	68442b48c8	HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495 )	2020-11-28 12:19:57 +08:00
Akira Ajisaka	2ce2198287	HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2020-11-26 11:32:10 +09:00
Akira Ajisaka	65002c9486	Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488 )" because JIRA ID is missing. This reverts commit `ce5b3d7975`.	2020-11-26 11:30:53 +09:00

... 10 11 12 13 14 ...

10185 Commits