hadoop

Author	SHA1	Message	Date
litao	e565b05c80	HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754 ). Contributed by tomscut.	2021-03-14 04:07:31 +05:30
lamberken	6921ec8b40	HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751 ). Contributed by Xie Lei. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-13 14:22:11 +08:00
Xiaoyu Yao	7cf5969f0b	HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764 )	2021-03-12 11:27:53 -08:00
Takanobu Asanuma	ecd3335187	HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.	2021-03-12 15:42:15 +09:00
Akira Ajisaka	23b343aed1	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753 ) Removed findbugs from the hadoop build images and added spotbugs instead. Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>	2021-03-11 10:56:07 +09:00
Hui Fei	6a55baeee4	HDFS-15875. Check whether file is being truncated before truncate (#2746 )	2021-03-10 14:11:29 +08:00
litao	6a922f9896	HDFS-15873. Add namenode address in logs for block report (#2743 )	2021-03-08 11:50:51 -05:00
He Xiaoqiao	4a0b7f7ebe	HDFS-15384. Document getLocatedBlocks(String src, long start) of DFSClient only return partial blocks. Contributed by Yang Yun.	2021-03-08 12:18:46 +08:00
tomscut	9cb51bf106	HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668 ) Contributed by tomscut.	2021-03-06 13:40:44 -08:00
litao	cdba06e380	HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739 ) Co-authored-by: tomscut <litao@bigo.sg>	2021-03-03 22:34:02 +09:00
JJiaguodong	8af56de1fa	HADOOP-17560. Fix some spelling errors (#2730 ) Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>	2021-03-03 11:41:05 +09:00
zhuqi	1f1a1ef52d	HDFS-15856: Make write pipeline retry times configurable. (#2721 ). Contributed by Qi Zhu Reviewed-by: Ayush Saxena <ayushsaxena@apache.org> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>	2021-03-02 13:16:11 +08:00
Konstantin V Shvachko	9501c698f4	HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.	2021-03-01 17:17:23 -08:00
litao	32353eb38a	HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718 ) Authored-by: tomscut <litao@bigo.sg>	2021-03-01 07:52:59 -08:00
S O'Donnell	c3b3b36dee	HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell	2021-03-01 11:36:41 +00:00
S O'Donnell	9c583f36d5	HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee	2021-02-24 10:29:36 +00:00
LeonGao	940c780feb	HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704 )	2021-02-23 22:29:48 -08:00
Ayush Saxena	647d23b611	HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.	2021-02-23 19:08:38 +05:30
Takanobu Asanuma	a57282540f	HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714 )	2021-02-23 15:12:06 +09:00
Wang Yuxuan	1e32ab7d2f	HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang	f8059d0bdf	Revert "first commit (#2559 )" This reverts commit `1b94569d89`.	2021-02-22 18:13:06 +08:00
Wang Yuxuan	1b94569d89	first commit (#2559 ) Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>	2021-02-22 02:10:39 -08:00
jianghuazhu	98ffd65d8b	HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686 ) Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>	2021-02-21 23:22:32 -08:00
Ayush Saxena	ff84a57483	HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.	2021-02-20 19:48:00 +05:30
Ayush Saxena	2ce5752fa8	HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun.	2021-02-20 19:34:29 +05:30
Takanobu Asanuma	25af8901a9	HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.	2021-02-19 14:44:58 +09:00
Akira Ajisaka	9a298d180d	Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 )" This reverts commit `4cf3531583`.	2021-02-19 11:09:10 +09:00
Hui Fei	2970bd93f3	HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694 )	2021-02-19 09:07:22 +08:00
Ayush Saxena	e391844e8e	HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.	2021-02-18 01:39:04 +05:30
Akira Ajisaka	0ed9e36282	HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696 )	2021-02-17 01:59:24 -05:00
Akira Ajisaka	4cf3531583	HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454 ) Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images, and added spotbugs in the images instead. Reviewed-by: Masatake Iwasaki <iwasakims@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org> Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>	2021-02-17 10:38:20 +09:00
Akira Ajisaka	2b3c5b1733	HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702 ) Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2021-02-16 14:50:50 +09:00
Zehao Chen	07a4220cd2	HDFS-15821. Add metrics for in-service datanodes (#2690 ). Contributed by Zehao Chen.	2021-02-15 11:14:32 -06:00
litao	c3134ab3a9	HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676 )	2021-02-14 00:12:24 -05:00
Steve Loughran	798df6d699	HADOOP-13327 Output Stream Specification. (#2587 ) This defines what output streams and especially those which implement Syncable are meant to do, and documents where implementations (HDFS; S3) don't. With tests. The file:// FileSystem now supports Syncable if an application calls FileSystem.setWriteChecksum(false) before creating a file -checksumming and Syncable.hsync() are incompatible. Contributed by Steve Loughran.	2021-02-10 10:28:59 +00:00
LeonGao	9434c1eccc	HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679 )	2021-02-09 10:49:28 +00:00
bshashikant	2df2dfb9ed	HDFS-15817. Rename snapshots while marking them deleted. (#2677 )	2021-02-09 10:27:51 +05:30
LeonGao	0e2b3086e3	HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625 )	2021-02-08 11:01:21 -08:00
bshashikant	19ae0faacc	HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682 )	2021-02-06 08:26:12 -08:00
Jim Brennan	c4918fb298	[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan	2021-02-05 21:20:30 +00:00
fermi	7ef2875c89	HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang	2021-02-05 11:01:49 +08:00
maobaolong	b99dba38b9	HDFS-15819. Fix import style #2681	2021-02-05 10:11:27 +08:00
Hui Fei	15a1f7adfc	HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang	2021-02-04 09:19:51 +08:00
Kihwal Lee	182623d2bc	HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.	2021-02-03 08:55:50 -06:00
Hui Fei	66ecee333e	HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang	2021-02-03 20:00:45 +08:00
YaYun-Wang	394b9f7a5c	HDFS-15624. fix the function of setting quota by storage type (#2377 ) 1. puts NVDIMM to the end of storage type enum to make sure compatibility. 2. adds check to make sure the software layout version is satisfied Co-authored-by: su xu <kevinbrandon@163.com> Co-authored-by: huangtianhua <huangtianhua223@gmail.com> Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Ayush Saxena <ayushsaxena@apache.org> Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>	2021-02-02 22:44:34 -08:00
crossfire	18978f2e20	HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657 )	2021-02-02 09:02:09 +00:00
He Xiaoqiao	1b893e10ae	HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-02 13:50:54 +08:00
belugabehr	21a3fc3d2d	HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633 )	2021-02-01 09:40:01 -08:00
He Xiaoqiao	b54134661b	HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.	2021-02-01 23:28:29 +08:00
Narges Shadab	115623a6ee	HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652 )	2021-02-01 01:56:59 -08:00
Siyao Meng	1a205cc3ad	HADOOP-17424. Replace HTrace with No-Op tracer (#2645 )	2021-02-01 13:42:44 +09:00
Gautham B A	ad483fd66e	HDFS-15740. Add x-platform utilities (#2567 )	2021-01-29 10:11:38 -08:00
Jim Brennan	f565476939	[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.	2021-01-28 18:17:15 +00:00
Jim Brennan	6184ab07f6	[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.	2021-01-28 16:47:52 +00:00
sunlisheng	f8769e0f4b	HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.	2021-01-28 10:10:39 +08:00
Ayush Saxena	e40f99f6d5	HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun.	2021-01-26 14:25:27 +05:30
Akira Ajisaka	45b3a84709	HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626 )	2021-01-21 10:07:46 +09:00
He Xiaoqiao	97f843de3a	HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.	2021-01-16 14:06:56 +08:00
LeonGao	7743d40ac5	HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583 ). Contributed by Leon Gao.	2021-01-15 16:28:11 -08:00
Masatake Iwasaki	630f8ddd2c	HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-14 16:19:35 +09:00
touchida	8ec824f2ba	HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598 ) Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2021-01-13 11:23:07 +09:00
Yiqun Lin	0d7ac54510	HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li.	2021-01-12 23:38:55 +08:00
Masatake Iwasaki	32abc0af49	HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507 )	2021-01-11 11:59:32 -08:00
He Xiaoqiao	ec22850dbe	Make upstream aware of 3.2.2 release.	2021-01-09 18:06:08 +08:00
Ahmed Hussein	77435a025e	HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601 ). Contributed by Ahmed Hussein and Daryn Sharp.	2021-01-08 13:10:09 -06:00
Ayush Saxena	2ba7ec2b48	HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun.	2021-01-08 16:27:54 +05:30
lfengnan	87bd4d2aca	HDFS-15754. Add DataNode packet metrics (#2578 ) Contributed by Fengnan Li.	2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang	2b4febcf57	HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533 )	2021-01-04 20:54:44 -08:00
Ayush Saxena	77299ae992	HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun.	2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang	66ee0a6df0	HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453 )	2021-01-04 09:43:58 -08:00
Masatake Iwasaki	2825d060cf	HDFS-15648. TestFileChecksum should be parameterized. (#2501 )	2021-01-04 14:00:57 +09:00
Ayush Saxena	1cd96e8dd8	HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.	2021-01-01 11:37:52 +05:30
maobaolong	3b77cf4446	HDFS-15749. Make size of editPendingQ can be configurable (#2572 )	2020-12-28 14:18:18 +08:00
hemanthboyina	16a20503ca	HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.	2020-12-22 17:27:31 +05:30
langlaile1221	4ae561bcb4	HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566 )	2020-12-22 09:25:11 +08:00
Ayush Saxena	ba496f61f8	HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.	2020-12-22 02:16:00 +05:30
Ayush Saxena	5ab1db9e21	HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut.	2020-12-21 09:58:30 +05:30
Ayush Saxena	2aea43bf4f	HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.	2020-12-20 23:06:19 +05:30
Ayush Saxena	3ecd3628d4	HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.	2020-12-20 15:18:25 +05:30
Ayush Saxena	70789fc6b9	HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.	2020-12-20 15:12:32 +05:30
Ahmed Hussein	c2672bb234	HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-17 11:33:29 -06:00
Ayush Saxena	0da00fc743	HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-12-14 20:42:23 -08:00
liuyanpunk	3363cc380b	HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534 )	2020-12-15 10:06:11 +05:30
S O'Donnell	9ed737001c	HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell	2020-12-11 20:53:10 +00:00
Ahmed Hussein	be35fa186c	HDFS-15717. Improve fsck logging. (#2529 ) Contributed by Kihwal Lee and Ahmed Hussein	2020-12-11 11:02:42 -06:00
Neil	9bd3c9bc50	HDFS-15720 namenode audit async logger should add some log4j config (#2532 )	2020-12-10 13:47:00 -08:00
Ahmed Hussein	3ec01b1bb3	HDFS-15711. Add Metrics to HttpFS Server. (#2521 ) Contributed by Ahmed Hussein and Kihwal Lee	2020-12-10 13:09:53 -06:00
Ahmed Hussein	01383a2172	HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528 )	2020-12-08 13:25:24 -06:00
Gautham B A	4ffec79b7c	HDFS-15712. Upgrade googletest to 1.10.0 (#2523 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-12-08 17:08:14 +09:00
crossfire	40f7543a6d	HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518 )	2020-12-07 15:49:45 -08:00
Ahmed Hussein	32099e36dd	HDFS-15707. NNTop counts don't add up as expected. (#2516 )	2020-12-07 14:22:06 -06:00
Yiqun Lin	da1ea2530f	HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.	2020-12-07 18:52:12 +08:00
Ayush Saxena	ad40715690	HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.	2020-12-07 15:00:19 +05:30
Ayush Saxena	7dda804a1a	HDFS-14090. RBF: Improved isolation for downstream name nodes. {Static}. Contributed by Fengnan Li and CR Hota.	2020-12-04 22:56:37 +05:30
Hui Fei	e2c1268ebd	HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.	2020-12-04 09:20:09 +08:00
Ahmed Hussein	07655a7040	HDFS-15706. HttpFS: Log more information on request failures. (#2515 )	2020-12-03 15:14:40 -08:00
Wei-Chiu Chuang	42a29199c0	HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.	2020-12-02 21:32:02 -08:00
LeonGao	6ff2409b31	HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483 ). Contributed by Leon Gao.	2020-12-02 15:53:09 -08:00
Ahmed Hussein	60201cbf69	HDFS-15703. Don't generate edits for set operations that are no-op (#2508 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 15:38:20 -06:00
Ahmed Hussein	2b5b556dd7	HDFS-15695. NN should not let the balancer run in safemode (#2489 ). Contributed by Daryn Sharp and Ahmed Hussein	2020-12-02 13:59:00 -06:00
Ahmed Hussein	918ba9e82f	HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487 ) Contributed by Kuhu Shukla and Ahmed Hussein	2020-12-01 15:02:53 -06:00
Masatake Iwasaki	6a1d7d9ed2	HDFS-15677. TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk. (#2503 )	2020-12-01 08:06:52 +09:00
Hui Fei	68442b48c8	HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495 )	2020-11-28 12:19:57 +08:00
Akira Ajisaka	2ce2198287	HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2020-11-26 11:32:10 +09:00
Akira Ajisaka	65002c9486	Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488 )" because JIRA ID is missing. This reverts commit `ce5b3d7975`.	2020-11-26 11:30:53 +09:00
Akira Ajisaka	ce5b3d7975	[JDK 11] Fix error in mvn package -Pdocs (#2488 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>	2020-11-26 11:27:30 +09:00
Siyao Meng	235947e282	HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472 )	2020-11-25 11:01:04 -08:00
He Xiaoqiao	641d8856d2	HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Hui Fei <ferhui@apache.org>	2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh	d73029463c	HDFS-15690. Add lz4-java as test dependency (#2481 )	2020-11-21 17:49:56 -08:00
Ahmed Hussein	747883af4d	HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443 ). Contributed by Ahmed Hussein.	2020-11-20 18:03:45 +05:30
Masatake Iwasaki	425996eb4a	HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467 )	2020-11-18 16:11:09 +09:00
Akira Ajisaka	b57f04cd5b	HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465 )	2020-11-17 10:56:56 +09:00
Takanobu Asanuma	f56cd88d7d	HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.	2020-11-13 11:02:26 +09:00
Issac Buenrostro	f5e6be337b	HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255 ) Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>	2020-11-12 14:57:26 -08:00
Peter Bacsko	cd0490e8c6	Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." This reverts commit `b0ab222a6c`.	2020-11-11 17:06:12 +01:00
Peter Bacsko	b0ab222a6c	YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.	2020-11-11 17:03:16 +01:00
jianghuazhu	e6d2dccbef	HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138 )	2020-11-10 16:34:17 -08:00
Siyao Meng	fbd2220167	HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448 )	2020-11-10 12:13:15 -08:00
Yizhong Zhang	82de07c794	HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.	2020-11-10 09:11:31 -08:00
Hui Fei	f1d5652359	HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang.	2020-11-10 13:09:03 +08:00
maobaolong	95c96605b3	HDFS-15667. Audit log record the unexpected allowed result when delete (#2437 )	2020-11-10 13:01:10 +08:00
LeonGao	9a9ab5b48e	HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288 ). Contributed by Leon Gao.	2020-11-09 15:06:16 -08:00
He Xiaoqiao	3067a25fa1	HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Yiqun Lin <yqlin@apache.org>	2020-11-04 13:53:46 +08:00
Ayush Saxena	7ac7840fc3	HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424 ). Contributed by Ayush Saxena.	2020-11-04 09:28:44 +05:30
Konstantin V Shvachko	d07dc7afb4	HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.	2020-11-03 12:02:40 -08:00
Ahmed Hussein	324879127a	HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419 )	2020-10-28 16:24:34 -07:00
Akira Ajisaka	b3ba74d72d	HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418 )	2020-10-28 19:12:23 +09:00
Ahmed Hussein	d0c786db4d	HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406 )	2020-10-27 18:56:40 -07:00
Ahmed Hussein	98097b8f19	HDFS-15457. TestFsDatasetImpl fails intermittently (#2407 )	2020-10-27 18:52:56 -07:00
Hui Fei	8a6d5b9151	HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416 )	2020-10-28 09:13:25 +08:00
Ahmed Hussein	ae74407ac4	HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 20:45:00 +09:00
Akira Ajisaka	3848b44b37	HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309 )	2020-10-27 19:18:08 +09:00
Ahmed Hussein	afaab3d332	HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404 ) Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>	2020-10-27 16:56:30 +09:00
Hui Fei	12c908c827	HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.	2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang	74634eb002	HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.	2020-10-23 12:05:53 -07:00
Inigo Goiri	e7aa4da0a2	HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.	2020-10-23 09:47:10 -07:00
Hui Fei	264c948e60	HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang	2020-10-23 14:06:22 +08:00
Kihwal Lee	da1b6e3cc2	HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.	2020-10-22 16:59:49 -05:00
Kihwal Lee	88a9f42f32	HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.	2020-10-21 00:59:35 -05:00
Takanobu Asanuma	30f06e0c74	HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394 )	2020-10-20 19:08:59 +09:00
Ayush Saxena	2e8cafac3b	HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.	2020-10-19 10:48:47 +05:30
Ayush Saxena	173310e2f5	HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.	2020-10-19 10:34:17 +05:30
Ayush Saxena	1e3a6efcef	HADOOP-17288. Use shaded guava from thirdparty. (#2342 ). Contributed by Ayush Saxena.	2020-10-17 12:01:18 +05:30
Kihwal Lee	740a2c4635	HDFS-15627. Log delete audits before collecting blocks.	2020-10-16 11:13:41 -05:00
Ayush Saxena	cc57eebe45	HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386 ). Contributed by Ayush Saxena.	2020-10-16 10:36:24 +05:30
Kihwal Lee	e45407128d	HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein.	2020-10-14 17:26:04 -05:00
Siyao Meng	a308a1ec22	HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370 )	2020-10-13 10:59:42 -07:00
Gabor Bota	59f01a548e	HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same Contributed by Gabor Bota.	2020-10-13 16:17:44 +01:00
Mukund Thakur	0507c4160f	HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur. Change-Id: I6cec41c20467920cf21f169653553535414b2680	2020-10-13 13:31:00 +01:00
Steve Loughran	d606ddc12c	Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 )" (reverting to change the JIRA number) This reverts commit `3e1b1dfd77`. Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd	2020-10-13 13:30:38 +01:00
Mukund Thakur	3e1b1dfd77	HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380 ) Fixes the regression caused by HADOOP-17281, where the WebHDFS client listStatusIterator (correctly) throws NoSuchElementException when next() runs out of values. Contributed by Mukund Thakur.	2020-10-13 13:27:55 +01:00
Konstantin V Shvachko	b3786d6c3c	HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.	2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G	b76b36ebbc	HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378 ). Contributed by Uma Maheswara Rao G.	2020-10-12 14:22:44 -07:00
Akira Ajisaka	69ef9b1ee8	HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375 )	2020-10-12 09:52:12 +09:00
LeonGao91	de055ec8aa	HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299 )	2020-10-10 22:08:36 +05:30
Hui Fei	518a212cff	HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363 )	2020-10-09 16:12:22 +08:00
Karthik Palanisamy	735e85a1d2	HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365 )	2020-10-08 12:51:04 +05:30
Karthik Palanisamy	df4006eb81	HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366 )	2020-10-07 22:09:04 +05:30
bshashikant	16aea11c94	HDFS-15611. Add list Snapshot command in WebHDFS. (#2355 )	2020-10-07 10:34:32 +05:30
Akira Ajisaka	074f0d46af	HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360 ) * Support getHAServiceState in DFSRouter	2020-10-07 13:37:45 +09:00
hemanthboyina	921ca1f554	HDFS-15543. RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled. Contributed by Hemanth Boyina.	2020-10-07 09:58:53 +05:30
Doroszlai, Attila	1cfe5916e2	HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-10-06 14:58:42 -07:00
Fei Hui	6ece6407e1	HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356 ). Contributed by Hui Fei.	2020-10-05 22:11:24 +05:30
Siyao Meng	43b0c0b054	HDFS-15607. Create trash dir when allowing snapshottable dir (#2352 )	2020-10-05 05:02:00 -07:00
huangtianhua	a89ca56a1b	HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347 )	2020-09-29 19:00:55 -07:00
Ayush Saxena	d686a655bc	HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui.	2020-09-29 00:03:11 +05:30
Ayush Saxena	fc8a6dd8a9	HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui.	2020-09-28 23:55:31 +05:30
Fei Hui	984875c624	HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345 )	2020-09-28 09:54:01 -07:00
Inigo Goiri	eae0035a2d	HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao.	2020-09-28 09:40:39 -07:00
Takanobu Asanuma	d89af79507	HDFS-15577. Refactor TestTracing. (#2302 ) Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2020-09-28 18:27:41 +09:00
Vinayakumar B	82b86e3754	HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng	2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G	899dea2a21	HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-25 21:21:01 -07:00
Ye Ni	00c4de63cf	HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332 )	2020-09-25 09:47:54 -07:00
Uma Maheswara Rao G	3ccc962b99	HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-24 07:07:48 -07:00
Aryan Gupta	486ddb73f9	HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330 ) Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>	2020-09-24 21:15:55 +09:00
huangtianhua	ff59fbb8b0	HDFS-15025. Applying NVDIMM storage media to HDFS (#2189 ) * HDFS-15025. Applying NVDIMM storage media to HDFS Co-authored-by: YaYun-Wang <yywangyayun@163.com> Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>	2020-09-24 14:27:04 +05:30
bshashikant	368f2f637e	HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326 )	2020-09-24 14:00:41 +05:30
Ye Ni	ae089f2db7	HDFS-15557. Log the reason why a storage log file can't be deleted (#2274 )	2020-09-22 13:23:20 -07:00
Hui Fei	474fa80bfb	HADOOP-17277. Correct spelling errors for separator (#2322 ) Contributed by Hui Fei.	2020-09-22 17:10:33 +01:00
Kihwal Lee	dfc2682213	HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross.	2020-09-22 10:53:04 -05:00
lfengnan	3e8b1e7426	HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266 ). Contributed by Fengnan Li.	2020-09-21 23:24:13 +05:30
Ayush Saxena	7a6265ac42	HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang.	2020-09-21 08:50:08 +05:30
Ayush Saxena	8eba922bb3	HDFS-15579. RBF: The constructor of PathLocation may got some misunderstanding. Contributed by Janus Chow.	2020-09-20 08:54:25 +05:30
Ayush Saxena	2377df38ad	HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.	2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G	7bba4c609c	HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312 ). Contributed by Uma Maheswara Rao G.	2020-09-18 15:18:10 +05:30
S O'Donnell	20a0e6278d	HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell	2020-09-17 14:11:42 +01:00
bshashikant	425f48799c	HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296 )	2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G	e4cb0d3514	HDFS-15578: Fix the rename issues with fallback fs enabled (#2305 ). Contributed by Uma Maheswara Rao G. Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>	2020-09-16 22:43:00 -07:00
Takanobu Asanuma	5c5b2ed7c7	HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.	2020-09-16 01:36:46 +09:00
hemanthboyina	f4ed9f3f91	HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.	2020-09-15 10:10:21 +05:30
He Xiaoqiao	924959088e	HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.	2020-09-15 12:05:04 +08:00
Siyao Meng	6d3312de47	HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258 )	2020-09-14 13:31:34 -07:00
Akira Ajisaka	c78d18023d	HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267 )	2020-09-14 11:34:52 +09:00
Uma Maheswara Rao G	d2779de3f5	HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298 ). Contributed by Uma Maheswara Rao G.	2020-09-12 17:06:39 -07:00
Mingliang Liu	f59f7f2175	HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell	2020-09-12 01:41:38 -07:00
hemanthboyina	2a6fdcedeb	HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.	2020-09-11 18:21:25 +05:30
imbajin	89428f142f	HDFS-15551. Tiny Improve for DeadNode detector (#2265 ) Contributed by imbajin. Reviewed-by: leosunli <lisheng.sun08@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-09-11 15:44:03 +08:00
1996fanrui	56ebabd426	HADOOP-17222. Create socket address leveraging URI cache (#2241 ) Contributed by fanrui. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-09-10 22:30:52 -07:00
zz	12a316cdf9	HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185 ). Contributed by Zhenzhao Wang. Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>	2020-09-10 21:20:32 -07:00
Siyao Meng	43c52d6495	HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir (#2295 ) Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc	2020-09-10 15:55:23 +05:30
He Xiaoqiao	1d6d0d8207	HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun. Co-authored-by: Chen Zhang <chzhang1987@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org> Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>	2020-09-09 20:09:13 +08:00
Uma Maheswara Rao G	ac7d4623ae	HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275 ) Contributed by Uma Maheswara Rao G.	2020-09-07 11:36:13 -07:00
Uma Maheswara Rao G	b3660d0147	HDFS-15529: getChildFilesystems should include fallback fs as well (#2234 ). Contributed by Uma Maheswara Rao G.	2020-09-03 11:06:20 -07:00
He Xiaoqiao	60de592a88	HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.	2020-08-31 14:49:12 +08:00
bshashikant	2d03209a06	HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion (#2251 )	2020-08-31 11:29:48 +05:30
Steve Loughran	c392d9022a	HDFS-15471. TestHDFSContractMultipartUploader failing (#2252 ) Contributed by Steve Loughran (Was: broken by Steve Loughran)	2020-08-28 15:39:09 +01:00
Vivek Ratnavel Subramanian	06793da100	HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory (#2230 )	2020-08-27 11:36:30 -07:00
Tsz-Wo Nicholas Sze	41182a9b6d	HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. (#2233 )	2020-08-27 14:54:52 +05:30
Stephen O'Donnell	2ffe00fc46	HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-08-26 23:05:32 -07:00
Takanobu Asanuma	ca8e7a7725	HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina.	2020-08-27 12:10:39 +09:00
hemanthboyina	6e618b6a7e	HDFS-15536. RBF: Clear Quota in Router was not consistent.	2020-08-26 13:03:08 +05:30
S O'Donnell	6a49bf9bff	HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.	2020-08-25 15:18:36 +01:00
He Xiaoqiao	a7830423c5	HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.	2020-08-24 21:16:00 +08:00
hemanthboyina	83fd15b412	HDFS-14504. Rename with Snapshots does not honor quota limit.	2020-08-23 16:10:47 +05:30
Chen Liang	f734455e5d	HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.	2020-08-20 15:25:10 -07:00
Ayush Saxena	4813a37023	HDFS-15535. RBF: Fix Namespace path to snapshot path resolution for snapshot API. Contributed by Ayush Saxena.	2020-08-20 23:54:35 +05:30
Uma Maheswara Rao G	dd013f2fdf	HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229 ). Contributed by Uma Maheswara Rao G.	2020-08-19 09:30:41 -07:00
Siyao Meng	b65e43fe38	HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS (#2220 )	2020-08-18 19:28:19 +09:00
bshashikant	092bfe7c8e	HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. (#2172 ) * HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. * Addressed review comments. * Rebased to latest trunk and added snapshotTrashRoot config check. * Addressed review comments. * Removede empty line added in SnapshotManager.Java. * Addressed whitespace issues.	2020-08-17 12:56:13 +05:30
Ayush Saxena	e3d1966f58	HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.	2020-08-15 15:07:58 +05:30
Tsz-Wo Nicholas Sze	b93dd7c281	HDFS-15519. Check inaccessible INodes in FsImageValidation. (#2224 )	2020-08-14 10:10:01 -07:00
bshashikant	15a76e8d65	HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. (#2219 )	2020-08-14 20:46:48 +05:30
Vivek Ratnavel Subramanian	cb50e3fcf7	HDFS-15496. Add UI for deleted snapshots (#2212 )	2020-08-13 22:36:15 +05:30
Aryan Gupta	e592ec5f8b	HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. (#2217 ) * HDFS-15518. Fixed String operationName = ListSnapshot. * HDFS-15518. Changed ListSnapshot to operationName. Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>	2020-08-13 00:09:38 +05:30
S O'Donnell	10716040a8	HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang	2020-08-12 09:02:47 +01:00
Tsz-Wo Nicholas Sze	11cec9ab94	HDFS-15523. Fix findbugs warnings from HDFS-15520. (#2218 )	2020-08-11 21:41:28 -07:00
Siyao Meng	3fd3aeb621	HDFS-15492. Make trash root inside each snapshottable directory (#2176 )	2020-08-11 08:52:16 -07:00
Gautham B A	909f1e82d3	HADOOP-17196. Fix C/C++ standard warnings (#2208 ) * Passing C/C++ standard flags -std is not cross-compiler friendly as not all compilers support all values. * Thus, we need to make use of the appropriate flags provided by CMake in order to specify the C/C++ standards. Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-08-11 16:35:08 +09:00
Uma Maheswara Rao G	8955a6ceb7	HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205 ) * HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false * Used LambdaTestUtils#intercept in test	2020-08-11 00:01:58 -07:00
Tsz-Wo Nicholas Sze	592127bdc2	HDFS-15520 Use visitor pattern to visit namespace tree (#2203 )	2020-08-11 12:30:30 +05:30
Akira Ajisaka	32895f4f7e	HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.	2020-08-11 13:49:56 +09:00
Akira Ajisaka	7938ebfb9d	HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.	2020-08-11 11:14:58 +09:00
Akira Ajisaka	9062814bab	HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module	2020-08-11 10:58:56 +09:00
Ayush Saxena	c2a17659d1	HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.	2020-08-08 14:33:53 +05:30
Ayush Saxena	975b6024dd	HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.	2020-08-07 22:19:17 +05:30
Takanobu Asanuma	1d5ccc790b	HDFS-15512. Remove smallBufferSize in DFSClient. (#2191 )	2020-08-06 22:47:12 +05:30
Mingliang Liu	58def7cecb	HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188 ) Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-08-04 20:48:45 -07:00
bshashikant	e072d33327	HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable (#2175 )	2020-08-04 14:10:29 +05:30
hemanthboyina	82f3ffcd64	HDFS-15503. File and directory permissions are not able to be modified from WebUI.	2020-08-03 23:21:01 +05:30
Akira Ajisaka	c40cbc57fa	HADOOP-17091. [JDK11] Fix Javadoc errors (#2098 )	2020-08-03 10:46:51 +09:00
bshashikant	d8a2df25ad	HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181 )	2020-08-02 21:59:56 +05:30
hemanthboyina	528a799a78	HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.	2020-08-02 12:02:37 +05:30
Yuan	e756fe3590	HDFS-14950. fix missing libhdfspp lib in dist-package (#1947 ) libhdfspp.{a,so} are missed in dist-package. This patch fixed this by copying these libs to the right directory Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>	2020-07-31 16:49:49 +09:00
Tsz-Wo Nicholas Sze	05b3337a46	HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots (#2165 )	2020-07-30 23:06:51 +05:30
bshashikant	68287371cc	HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166 )	2020-07-29 21:33:25 +05:30
touchida	026dce5334	HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. (#2135 )	2020-07-27 09:55:11 -07:00
bshashikant	2d12496643	HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (#2163 )	2020-07-22 16:16:27 -07:00
Uma Maheswara Rao G	ac9a07b51a	HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160 ). Contributed by Uma Maheswara Rao G.	2020-07-21 23:29:10 -07:00
Masatake Iwasaki	1b29c9bfee	HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155 )	2020-07-22 13:40:20 +09:00
Inigo Goiri	8b7695bb26	HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.	2020-07-21 09:00:07 -07:00
Tsz-Wo Nicholas Sze	d57462f2da	HDFS-15479. Ordered snapshot deletion: make it a configurable feature (#2156 )	2020-07-21 11:36:24 +05:30
Shashikant Banerjee	d9441f95c3	HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.	2020-07-21 11:13:05 +05:30
Chen Liang	3833c616e0	HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.	2020-07-20 12:49:58 -07:00
Ayush Saxena	6cbd8854ee	HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.	2020-07-20 21:49:05 +05:30
Tsz-Wo Nicholas Sze	2cec50cf16	HDFS-15463. Add a tool to validate FsImage. (#2140 )	2020-07-20 11:44:30 +05:30
Suraj Naik	4101b0c0ed	HDFS-15476 Make AsyncStream executor private (#2151 )	2020-07-19 23:26:54 +05:30
Ayush Saxena	8a9a674ef1	HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu.	2020-07-18 15:36:55 +05:30
Ayush Saxena	6bcb24d269	HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.	2020-07-18 14:33:43 +05:30
Ahmed Hussein	1f71c4ae71	HADOOP-17099. Replace Guava Predicate with Java8+ Predicate Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 11:39:06 -05:00
Ahmed Hussein	98fcffe93f	HADOOP-17101. Replace Guava Function with Java8+ Function Signed-off-by: Jonathan Eagles <jeagles@gmail.com>	2020-07-15 09:53:18 -05:00
Shashikant Banerjee	85d4718ed7	HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.	2020-07-15 10:32:25 +05:30
Gautham B A	cce5a6f609	HDFS-15385 Upgrade boost library to 1.72 (#2051 ) * Removed the asio-1.10.2 that was checked in under libhdfspp/third_party directory.	2020-07-15 13:39:48 +09:00
zhaoyim	bdce75d737	HDFS-15371. Nonstandard characters exist in NameNode.java (#2032 ) Contributed by zhaoyim	2020-07-14 14:42:12 +01:00
Steve Loughran	b9fa5e0182	HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. Contributed by Steve Loughran. Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3	2020-07-13 13:30:02 +01:00
He Xiaoqiao	b97fea65e7	HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.	2020-07-13 14:12:48 +08:00
lfengnan	84b74b335c	HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110 )	2020-07-12 00:10:12 -07:00
Uma Maheswara Rao G	3e70006639	HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132 ). Contributed by Uma Maheswara Rao G.	2020-07-11 23:50:04 -07:00
Xiaoyu Yao	f91a8ad88b	HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085 )	2020-07-09 11:33:37 -07:00
He Xiaoqiao	4f26454a7d	HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang.	2020-07-08 00:02:39 +08:00
Ye Ni	f77bbc2123	HDFS-15312. Apply umask when creating directory by WebHDFS (#2096 )	2020-07-06 19:01:46 -07:00
Uma Maheswara Rao G	dc0626b5f2	HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.	2020-07-06 18:50:03 -07:00
Ye Ni	e820baa6e6	HDFS-15417. RBF: Get the datanode report from cache for federation WebHDFS operations (#2080 )	2020-07-06 16:17:09 -07:00
Shanyu Zhao	834372f404	HDFS-15451. Do not discard non-initial block report for provided storage. (#2119 ). Contributed by Shanyu Zhao. Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>	2020-07-06 23:43:34 +08:00
Uma Maheswara Rao G	55a2ae80dc	HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.	2020-07-04 13:45:49 -07:00
Ayush Saxena	f86f15cf20	HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.	2020-07-04 13:00:43 +05:30
Uma Maheswara Rao G	1f2a80b5e5	HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-07-04 00:12:10 -07:00
He Xiaoqiao	9ac498e300	HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.	2020-07-01 12:38:07 +08:00
Stephen O'Donnell	2a67e2b1a0	HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-30 07:10:17 -07:00
Akira Ajisaka	c71ce7ac33	HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.	2020-06-28 16:02:47 +09:00
Ayush Saxena	8db38c98a6	HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.	2020-06-27 19:13:01 +05:30
Virajith Jalaparti	bed0a3a374	HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100 ) * HDFS-15436. Default mount table name used by ViewFileSystem should be configurable * Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests * Address Uma's comments on PR#2100 * Sort lists in test to match without concern to order * Address comments, fix checkstyle and fix failing tests * Fix checkstyle	2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G	d5e1bb6155	HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.	2020-06-26 01:29:38 -07:00
lfengnan	84110d850e	HDFS-15383. RBF: Add support for router delegation token without watch (#2047 ) Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.	2020-06-23 13:12:29 -07:00
Uma Maheswara Rao G	7c02d1889b	HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.	2020-06-23 01:42:25 -07:00
Ayush Saxena	17ffcab5f6	HDFS-14546. Document block placement policies. Contributed by Amithsha.	2020-06-21 17:27:28 +05:30
Uma Maheswara Rao G	b27810aa60	HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.	2020-06-20 00:32:02 -07:00
Stephen O'Donnell	d50e93ce7b	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang	edf716a5c3	Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." This reverts commit `730a39d138`.	2020-06-18 06:45:15 -07:00
S O'Donnell	123777823e	HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina	2020-06-18 12:26:22 +01:00
Yiqun Lin	9cbd76cc77	HDFS-15346. FedBalance tool implementation. Contributed by Jinglun.	2020-06-18 13:33:25 +08:00
Ayush Saxena	c8ed33cd2a	HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.	2020-06-17 13:55:40 +05:30
Stephen O'Donnell	730a39d138	HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-06-16 15:59:13 -07:00
Eric Yang	81d8a887b0	SPNEGO TLS verification Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-06-15 10:56:16 +09:00
Takanobu Asanuma	f41a144077	HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.	2020-06-15 09:15:53 +09:00
Inigo Goiri	719b53a79d	HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.	2020-06-13 09:35:05 -07:00
Uma Maheswara Rao G	785b1def95	HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.	2020-06-12 14:32:19 -07:00
Vinayakumar B	e154084770	HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026 )	2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G	93b121a971	HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.	2020-06-10 15:00:02 -07:00
Ayush Saxena	b735a77717	HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.	2020-06-10 12:06:16 +05:30
Ayush Saxena	635e6a16d0	HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi.	2020-06-10 11:50:37 +05:30
Ayush Saxena	8525874561	HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. *Added missed test file.	2020-06-09 18:53:33 +05:30
Uma Maheswara Rao G	3ca15292c5	HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.	2020-06-06 08:11:57 -07:00
Ayush Saxena	cc671b16f7	HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena	2020-06-05 22:19:38 -07:00
Uma Maheswara Rao G	76fa0222f0	HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.	2020-06-05 10:58:21 -07:00
Ayush Saxena	2326123705	HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.	2020-06-05 19:15:05 +05:30
Toshihiro Suzuki	545a0a147c	HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052 ) Contributed by Toshihiro Suzuki.	2020-06-05 11:11:49 +01:00
Ayush Saxena	e8cb2ae409	HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.	2020-06-03 12:47:15 +05:30
Ayush Saxena	f6453244ab	HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan.	2020-06-03 12:16:36 +05:30
Uma Maheswara Rao G	ed83c865dd	HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.	2020-06-02 11:09:26 -07:00
Ayush Saxena	ae13a5ccbe	HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.	2020-05-31 17:10:59 +05:30
Wanqiang Ji	d9838f2d42	HADOOP-17055. Remove residual code of Ozone (#2039 )	2020-05-29 16:49:18 +09:00
Ayush Saxena	a838d871a7	HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He.	2020-05-28 20:52:58 +05:30
Ayush Saxena	9b38be43c6	HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.	2020-05-28 13:00:04 +05:30
Inigo Goiri	2148a8fe64	HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina.	2020-05-27 11:06:13 -07:00
Ayush Saxena	6c9f75cf16	HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena.	2020-05-26 16:26:57 +05:30
Ayush Saxena	f43a152b97	HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun.	2020-05-24 18:03:41 +05:30
Ayush Saxena	f4901d0778	HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun.	2020-05-24 17:59:16 +05:30
Inigo Goiri	6e04b00df1	HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.	2020-05-23 09:58:19 -07:00
Ayush Saxena	e0ae232f66	HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.	2020-05-23 18:38:32 +05:30
Ayush Saxena	f99fcb26ab	HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.	2020-05-23 18:29:31 +05:30
Takanobu Asanuma	4d22d1c58f	HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.	2020-05-23 17:28:38 +09:00
Uma Maheswara Rao G	4734c77b4b	HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.	2020-05-21 21:34:58 -07:00
leosunli	be374faf42	HDFS-13639. SlotReleaser is not fast enough (#1885 )	2020-05-21 13:21:17 -07:00
Yiqun Lin	1983eea62d	HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun.	2020-05-20 10:39:40 +08:00
Abhishek Das	ce4ec74453	HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.	2020-05-18 22:27:12 -07:00
Chen Liang	7bb902bc0d	HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang	2020-05-18 10:58:52 -07:00
Ayush Saxena	c84e6beada	HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena.	2020-05-18 22:24:34 +05:30
Wei-Chiu Chuang	2abcf7762a	HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 )	2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang	4525292d41	Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 )" This reverts commit `86e6aa8eec`.	2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang	50caba1a92	HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun.	2020-05-18 08:40:38 -07:00
He Xiaoqiao	a3f44dacc1	HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-05-18 07:09:43 -07:00
pustota2009	86e6aa8eec	HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016 ) Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.	2020-05-18 07:04:04 -07:00
Ayush Saxena	a3809d2023	HDFS-15082. RBF: Check each component length of destination path when add/update mount entry. Contributed by Xiaoqiao He.	2020-05-17 19:45:34 +05:30
Ayush Saxena	6e416a83d1	HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.	2020-05-17 03:06:27 +05:30
Ayush Saxena	178336f8a8	HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He.	2020-05-16 16:35:06 +05:30
Uma Maheswara Rao G	ac4a2e11d9	HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.	2020-05-14 17:29:35 -07:00
Surendra Singh Lilhore	1958cb7c2b	YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng	2020-05-14 15:09:34 +05:30
Ayush Saxena	0918433b4d	YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng.	2020-05-14 00:36:20 +05:30
Xiaoyu Yao	3cacf1ce56	HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. (#2004 )	2020-05-13 11:47:19 -07:00
Surendra Singh Lilhore	743c2e9071	HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina	2020-05-13 15:01:07 +05:30
Akira Ajisaka	8ffc356b1e	Revert "SPNEGO TLS verification" This reverts commit `ba66f3b454`.	2020-05-13 17:14:14 +09:00
Ayush Saxena	936bf09c37	HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq.	2020-05-12 21:54:54 +05:30
Elixir Kook	a3f945fb84	HADOOP-17035. fixed typos (timeout, interruped) (#2007 ) Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>	2020-05-12 10:50:04 -05:00
Xiaoyu Yao	047d8879e7	HDFS-15345. RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442.	2020-05-12 08:31:04 -07:00
S O'Donnell	29dddb8a14	HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.	2020-05-12 15:07:51 +01:00
Takanobu Asanuma	928b81a533	HDFS-15350. Set dfs.client.failover.random.order to true as default. (#2008 )	2020-05-12 09:04:03 -05:00
Ayush Saxena	8dad38c0be	HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.	2020-05-12 18:34:26 +05:30
Ayush Saxena	0fe49036e5	HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.	2020-05-12 13:11:31 +05:30
Ayush Saxena	4c53fb9ce1	HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.	2020-05-11 16:48:34 +05:30
Ayush Saxena	aab9e0b16e	HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest.	2020-05-10 11:43:30 +05:30
Ayush Saxena	cb64e993c2	HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi.	2020-05-10 00:11:46 +05:30
Ayush Saxena	d59de27c01	HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.	2020-05-08 00:53:49 +05:30
Inigo Goiri	130f89e068	HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.	2020-05-06 13:18:32 -07:00
Ayush Saxena	7fddf4855e	HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.	2020-05-06 16:55:04 +05:30
Wei-Chiu Chuang	adecdb8b53	HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998 ) Reviewed-by: Arpit Agarwal <arp@apache.org>	2020-05-05 16:04:54 -07:00
Uma Maheswara Rao G	9c8236d04d	HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.	2020-05-04 17:55:40 -07:00
Babneet Singh	1996351b0b	HDFS-15270. Account for env == NULL in hdfsThreadDestructor (#1951 ) OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread termination, accessing JNIEnv in hdfsThreadDestructor (env)->GetJavaVM, yields a SIGSEGV since env is NULL after thread cleanup is performed. The main purpose of hdfsThreadDestructor is to invoke DetachCurrentThread, which performs thread cleanup in OpenJ9. Since OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked, hdfsThreadDestructor should account for env == NULL and skip DetachCurrentThread. Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>	2020-05-04 13:08:53 -07:00
Konstantin V Shvachko	ebb878bab9	HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.	2020-05-04 10:29:50 -07:00
Ayush Saxena	8dace8ff3a	HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi.	2020-05-03 23:59:17 +05:30
Ayush Saxena	d6fc482a54	Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.	2020-05-03 19:09:14 +05:30
Mingliang Liu	44de193bec	HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li	2020-05-02 20:56:29 -07:00
Akira Ajisaka	f40dacdc2e	HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992 ) Signed-off-by: Mingliang Liu <liuml07@apache.org>	2020-05-02 13:02:27 -07:00
Eric Yang	ba66f3b454	SPNEGO TLS verification Signed-off-by: Akira Ajisaka <aajisaka@apache.org>	2020-05-02 21:50:13 +09:00
Shashikant Banerjee	82343790ee	HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.	2020-05-01 12:16:02 +05:30
Mingliang Liu	263c76b678	HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>	2020-04-30 14:15:28 -07:00
Akira Ajisaka	b5b45c53a4	HDFS-15309. Remove redundant String.valueOf method on ExtendedBlockId.java. Contributed by bianqi.	2020-04-30 15:46:46 +09:00
Inigo Goiri	31b2f687ef	HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina.	2020-04-29 13:41:36 -07:00
S O'Donnell	9ca6298a9a	HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.	2020-04-29 15:56:06 +01:00
Surendra Singh Lilhore	db6252b6c3	HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore.	2020-04-29 10:58:35 +05:30
Aryan Gupta	816042e62b	HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980 )	2020-04-29 09:19:59 +05:30
Akira Ajisaka	5e0eda5d5f	HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.	2020-04-28 06:17:14 +09:00
Toshihiro Suzuki	62c26b91fd	HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 (#1979 )	2020-04-27 17:20:47 +09:00
Mingliang Liu	74d4a18735	HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena	2020-04-25 15:43:02 -07:00
Dhiraj	453c9c9a67	HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964 ) Contributed by Dhiraj Hegde. Signed-off-by: Mingliang Liu <liuml07@apache.org> Signed-off-by: Inigo Goiri <inigoiri@apache.org>	2020-04-25 13:04:32 -07:00
Wei-Chiu Chuang	453771f170	HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.	2020-04-24 11:53:24 -07:00
Mingliang Liu	5958af4dbc	HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina	2020-04-22 13:31:02 -07:00
Akira Ajisaka	e069a06137	HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky	2020-04-20 08:41:18 -07:00
Takanobu Asanuma	1fdfaebd98	HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina.	2020-04-20 23:09:33 +09:00
Toshihiro Suzuki	1824aee9da	HDFS-15217 Add more information to longest write/read lock held log	2020-04-18 13:52:07 -07:00
Surendra Singh Lilhore	8e6227441a	HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore.	2020-04-18 20:07:21 +05:30
Ayush Saxena	37d6582223	HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.	2020-04-17 13:32:02 +05:30
Stephen O'Donnell	3481895f8a	HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-04-16 20:18:25 -07:00
Ayush Saxena	cc5c1da7c1	HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang.	2020-04-15 23:01:58 +05:30
Ayush Saxena	8d49229c37	HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.	2020-04-12 12:05:50 +05:30
Ayush Saxena	275c478330	HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena.	2020-04-09 23:44:36 +05:30
Wei-Chiu Chuang	061afcdf30	HDFS-15269. NameNode should check the authorization API version only … (#1945 ) Reviewed-by: Takanobu Asanuma <tasanuma@apache.org> Reviewed-by: Akira Ajisaka <aajisaka@apache.org>	2020-04-09 09:19:35 -07:00
Ayush Saxena	1189af4746	HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena.	2020-04-08 15:55:14 +05:30
Toshihiro Suzuki	c12ddbd1de	HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922 )	2020-04-07 14:38:09 +09:00
Inigo Goiri	0b855b9f35	HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun.	2020-04-06 13:05:33 -07:00
Akira Ajisaka	1695d8d59c	HDFS-15258. RBF: Mark Router FSCK unstable. (#1934 )	2020-04-04 00:13:41 +09:00
Takanobu Asanuma	736659e0e1	HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina.	2020-04-02 19:21:02 +09:00
Ayush Saxena	4a3eb10972	HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He.	2020-04-02 11:06:01 +05:30
He Xiaoqiao	d3b5951572	HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2020-04-01 16:36:46 -07:00
Inigo Goiri	80b877a72f	HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li.	2020-03-30 12:29:21 -07:00
Ayush Saxena	960c9ebaea	HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang.	2020-03-30 02:40:25 +05:30
Brahma Reddy Battula	8914cf9167	Preparing for 3.4.0 development	2020-03-29 23:24:25 +05:30
Inigo Goiri	f7a94ec0a4	HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina.	2020-03-29 08:17:02 -07:00
Ayush Saxena	696a663cd0	HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest.	2020-03-29 00:01:22 +05:30
Chen Liang	f531a4a487	HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.	2020-03-27 12:55:38 -07:00
Ayush Saxena	cbe71eaddf	HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq.	2020-03-27 13:42:06 +05:30
Inigo Goiri	679631b188	HDFS-13470. RBF: Add Browse the Filesystem button to the UI.	2020-03-26 08:23:18 -07:00
Wei-Chiu Chuang	0fa7bf47df	HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909 )	2020-03-25 16:03:26 -07:00
Inigo Goiri	cdcb77a2c5	HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He.	2020-03-25 11:30:54 -07:00
Arpit Agarwal	a700803a18	HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348	2020-03-25 10:28:30 -07:00
Toshihiro Suzuki	d353b30baf	HDFS-15215. The Timestamp for longest write/read lock held log is wrong	2020-03-24 14:50:15 -07:00
Isa Hekmatizadeh	d9c4f1129c	HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902 ). Contributed by zhengchenyu.	2020-03-24 23:17:22 +05:30
Ayush Saxena	ea87d60493	HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun.	2020-03-24 22:56:09 +05:30
Karthik Palanisamy	5250cd6db3	HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870 )	2020-03-24 15:15:46 +05:30
He Xiaoqiao	5eddc82fb8	HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-03-23 22:41:31 -07:00
Akira Ajisaka	f59f6891c8	HDFS-15232. Fix libhdfspp test failures with GCC 7. (#1906 )	2020-03-24 11:36:14 +09:00
He Xiaoqiao	af64ce2f4a	HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-03-23 12:43:51 -07:00
Ayush Saxena	62c8015683	HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena.	2020-03-22 11:44:18 +05:30
Takanobu Asanuma	b74e47e31a	HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.	2020-03-22 01:44:55 +09:00
Ayush Saxena	f9bb2a8cc5	HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.	2020-03-19 22:26:34 +05:30
Ayush Saxena	bb41ddaf1e	HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.	2020-03-19 21:23:13 +05:30
Ayush Saxena	654db35fa2	HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.	2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang	096533c2dc	HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. Contributed by Wei-Chiu Chuang. Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d	2020-03-18 13:27:13 +00:00
Surendra Singh Lilhore	1d772dc542	HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.	2020-03-15 20:44:32 +05:30
Ayush Saxena	74780c22eb	HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang.	2020-03-15 16:30:39 +05:30
Ayush Saxena	8a7c54995a	HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina.	2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang	7b637a3341	HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.	2020-03-13 19:01:23 -07:00
He Xiaoqiao	e9955bb8ff	HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Reviewed-by: Brahma Reddy Battula <brahma@apache.org> Reviewed-by: Inigo Goiri <inigoiri@apache.org>	2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang	4b95c242ec	HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829 ) Reviewed-by: Xiaoyu Yao <xyao@apache.org>	2020-03-13 11:29:44 -07:00
Ayush Saxena	fb1d7635ae	HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang.	2020-03-13 21:10:03 +05:30
S O'Donnell	25497799cb	HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.	2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang	20903f72b4	HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun.	2020-03-12 19:23:12 -07:00
Inigo Goiri	0695f7a538	HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang.	2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore	f736408a83	HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.	2020-03-12 18:59:17 +05:30
Steve Loughran	c734d69a55	HADOOP-16898. Batch listing of multiple directories via an (unstable) interface Contributed by Steve Loughran. This moves the new API of HDFS-13616 into a interface which is implemented by HDFS RPC filesystem client (not WebHDFS or any other connector) This new interface, BatchListingOperations, is in hadoop-common, so applications do not need to be compiled with HDFS on the classpath. They must cast the FS into the interface. instanceof can probe the client for having the new interface -the patch also adds a new path capability to probe for this. The FileSystem implementation is cut; tests updated as appropriate. All new interfaces/classes/constants are marked as @unstable. Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681	2020-03-09 14:51:16 +00:00
Steve Loughran	d4d4c37810	HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden Contributed by Steve Loughran. Not all stores do complete validation here; in particular the S3A Connector does not: checking up the entire directory tree to see if a path matches is a file significantly slows things down. This check does take place in S3A mkdirs(), which walks backwards up the list of parent paths until it finds a directory (success) or a file (failure). In practice production applications invariably create destination directories before writing 1+ file into them -restricting check purely to the mkdirs() call deliver significant speed up while implicitly including the checks. Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0	2020-03-09 14:44:28 +00:00
Inigo Goiri	4062217189	HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena.	2020-03-06 10:14:53 -08:00
Inigo Goiri	3afd4cbe89	HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina.	2020-03-04 10:13:23 -08:00
Inigo Goiri	1d4d0fcbe1	HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.	2020-03-04 10:02:54 -08:00
Akshay Nehe	bbd704bb82	HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871 )	2020-03-04 11:07:31 +09:00
Xiaoyu Yao	0dd8956f2e	HADOOP-16885. Encryption zone file copy failure leaks a temp file Contributed by Xiaoyu Yao. Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails Copying file through the FsShell command into an HDFS encryption zone where the caller lacks permissions is leaks a temp ._COPYING file and potentially a wrapped stream unclosed. This is a convergence of a fix for S3 meeting an issue in HDFS. S3: a HEAD against a file can cache a 404, -you must not do any existence checks, including deleteOnExit(), until the file is written. Hence: HADOOP-16490, only register files for deletion the create worked and the upload is not direct. HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on create() failures. Which means that an entry is retained on the NN. -you need to register a file with deleteOnExit() even if the file wasn't created. This patch: * Moves the deleteOnExit to ensure the created file get deleted cleanly. * Fixes HDFS to close the wrapped stream on failures.	2020-03-02 13:22:00 +00:00
Ayush Saxena	1a636da041	HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun.	2020-02-29 09:55:29 +05:30
Inigo Goiri	97b797c314	HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.	2020-02-28 18:47:22 -08:00
Takanobu Asanuma	9a3a28018a	HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina.	2020-02-29 10:30:26 +09:00
Surendra Singh Lilhore	fccfb02672	HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.	2020-02-28 18:33:06 +05:30
Ayush Saxena	429da635ec	HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong.	2020-02-28 00:31:55 +05:30
Ayush Saxena	cd2c6b1aac	HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest.	2020-02-27 22:19:35 +05:30
Istvan Fajth	57aa048516	HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739 )	2020-02-27 07:48:14 -08:00
Surendra Singh Lilhore	2059f255d3	HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena.	2020-02-27 14:08:42 +05:30
Ayush Saxena	033a3d7ff9	HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.	2020-02-27 04:02:26 +05:30
Ayush Saxena	209630472a	HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun.	2020-02-27 02:22:24 +05:30
Kihwal Lee	352a4ec16d	HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein.	2020-02-26 09:33:29 -06:00
Stephen O'Donnell	900430b990	HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang	1c5d2f1fdc	HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.	2020-02-24 13:39:16 -08:00
Ayush Saxena	7f8685f476	HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena.	2020-02-24 20:38:04 +05:30
Ayush Saxena	93b8f453b9	HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li.	2020-02-24 20:17:01 +05:30
Ayush Saxena	b5698e0c33	HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun.	2020-02-24 00:07:18 +05:30
Ayush Saxena	9eb7a8bdf8	HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi.	2020-02-23 14:25:39 +05:30
Ayush Saxena	ba9025c7cd	HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun.	2020-02-22 14:27:26 +05:30
Konstantin V Shvachko	ecbcb058b8	HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko.	2020-02-21 17:53:37 -08:00
Inigo Goiri	ed70c115a8	HDFS-15172. Remove unnecessary deadNodeDetectInterval in DeadNodeDetector#checkDeadNodes(). Contributed by Lisheng Sun.	2020-02-21 16:36:30 -08:00
Konstantin V Shvachko	6f84269bcd	HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko.	2020-02-21 10:51:14 -08:00
Masatake Iwasaki	2338d25dc7	HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758 )	2020-02-21 11:56:07 +09:00
Inigo Goiri	ec7507162c	HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham.	2020-02-19 11:33:58 -08:00
Ayush Saxena	ac4b556e2d	HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena.	2020-02-19 08:20:59 +05:30
Ayush Saxena	84f7638840	HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri.	2020-02-18 00:43:33 +05:30
xiaoxiaopan118	439d935e1d	HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849 )	2020-02-17 15:55:10 +09:00
Surendra Singh Lilhore	810783d443	HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree.	2020-02-16 12:15:03 +05:30
Ayush Saxena	c75756fe13	HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena.	2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore	a98352ced1	HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina.	2020-02-13 16:57:41 +05:30
Akira Ajisaka	0ddb5f0881	HDFS-13989. RBF: Add FSCK to the Router (#1832 ) Co-authored-by: Inigo Goiri <inigoiri@apache.org>	2020-02-13 10:06:07 +09:00
Ayush Saxena	f09710bbb8	HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun	2020-02-12 20:29:35 +05:30
Ayush Saxena	3df0adaaea	HDFS-15127. RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Contributed by Inigo Goiri	2020-02-12 19:41:04 +05:30
Kihwal Lee	9b8a78d97b	HDFS-14758. Make lease hard limit configurable and reduce the default. Contributed by hemanthboyina.	2020-02-11 12:40:00 -06:00
Stephen O'Donnell	d7c136b9ed	HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>	2020-02-11 08:00:15 -08:00
Ayush Saxena	6191d4b4a0	HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun.	2020-02-09 23:32:22 +05:30
Ayush Saxena	d23317b102	HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang	2020-02-08 10:33:57 +05:30
Vinayakumar B	7dac7e1d13	HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635 ). Contributed by Vinayakumar B.	2020-02-07 14:51:24 +05:30
Chen Liang	ce7b8b5634	HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.	2020-02-04 12:12:35 -08:00
Kihwal Lee	10a60fbe20	HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan.	2020-02-04 12:22:35 -06:00
Stephen O'Donnell	1e3a0b0d93	HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> Co-authored-by: Akira Ajisaka <aajisaka@apache.org>	2020-01-31 16:13:02 -08:00
Kihwal Lee	799d4c1cf4	HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails intermittently. Contributed by Ahmed Hussein.	2020-01-29 11:00:27 -06:00
Chen Liang	483397c7f7	[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang	2020-01-28 15:20:36 -08:00
Chen Liang	3e86807802	Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang." This reverts commit `ff8ff0f7e5`.	2020-01-28 15:19:47 -08:00
Chen Liang	ff8ff0f7e5	[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang.	2020-01-28 15:14:58 -08:00
Inigo Goiri	1839c467f6	HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein.	2020-01-28 10:10:35 -08:00
Inigo Goiri	061421fc6d	HDFS-15145. HttpFS: getAclStatus() returns permission as null. Contributed by hemanthboyina.	2020-01-28 10:04:38 -08:00
Ayush Saxena	87c198468b	HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun.	2020-01-28 23:09:38 +05:30
Ayush Saxena	f876dc228b	HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena.	2020-01-28 19:34:21 +05:30
Ayush Saxena	6d008c0d39	HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest	2020-01-25 11:47:29 +05:30
Kihwal Lee	d10f77e3c9	HDFS-15119. Allow expiration of cached locations in DFSInputStream. Contributed by Ahmed Hussein.	2020-01-24 09:15:27 -06:00
Ayush Saxena	92c58901d7	HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena	2020-01-23 18:18:34 +05:30
Kihwal Lee	bd03053ea2	HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein.	2020-01-22 09:14:38 -06:00
Inigo Goiri	8cfc3673dc	HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui.	2020-01-21 13:29:20 -08:00
Inigo Goiri	b657822b98	HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein.	2020-01-21 13:22:53 -08:00
Akira Ajisaka	f6d20daf40	HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li.	2020-01-21 18:03:24 +09:00
Akira Ajisaka	1defe3a65a	HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.	2020-01-21 10:58:32 +09:00
Inigo Goiri	263413e838	HDFS-15112. RBF: Do not return FileNotFoundException when a subcluster is unavailable.	2020-01-16 10:39:14 -08:00
Vinayakumar B	edbbc03ce7	HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. (#1803 )	2020-01-16 23:27:50 +05:30
Chao Sun	d7c4f8ab21	HDFS-13616. Batch listing of multiple directories (#1725 )	2020-01-15 17:22:47 -08:00
Wei-Chiu Chuang	6b86a5110e	HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu.	2020-01-13 10:48:53 -08:00
Vinayakumar B	52b360a928	HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. (#1801 )	2020-01-13 10:40:29 +05:30
Ayush Saxena	cebce0a348	HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena.	2020-01-11 10:29:11 +05:30
Ayush Saxena	7b62409ace	HDFS-15108. RBF: MembershipNamenodeResolver should invalidate cache incase of active namenode update. Contributed by Ayush Saxena.	2020-01-11 10:21:31 +05:30
Chen Liang	26a969ec73	HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.	2020-01-10 17:06:59 -08:00

... 8 9 10 11 12 ...

9591 Commits