big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,276 Commits 2 Branches 2 Tags 264 MiB
feee41aa00
Commit Graph

9360 Commits

Author SHA1 Message Date
Akira Ajisaka
2b3c5b1733
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-02-16 14:50:50 +09:00
Zehao Chen
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
litao
c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676) 2021-02-14 00:12:24 -05:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
LeonGao
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 2021-02-09 10:49:28 +00:00
bshashikant
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. (#2677) 2021-02-09 10:27:51 +05:30
LeonGao
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625) 2021-02-08 11:01:21 -08:00
bshashikant
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682) 2021-02-06 08:26:12 -08:00
Jim Brennan
c4918fb298 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 21:20:30 +00:00
fermi
7ef2875c89 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 2021-02-05 11:01:49 +08:00
maobaolong
b99dba38b9
HDFS-15819. Fix import style #2681 2021-02-05 10:11:27 +08:00
Hui Fei
15a1f7adfc HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 2021-02-04 09:19:51 +08:00
Kihwal Lee
182623d2bc HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 2021-02-03 08:55:50 -06:00
Hui Fei
66ecee333e HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang 2021-02-03 20:00:45 +08:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
crossfire
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 2021-02-02 09:02:09 +00:00
He Xiaoqiao
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:50:54 +08:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
He Xiaoqiao
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:28:29 +08:00
Narges Shadab
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 2021-02-01 01:56:59 -08:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
Gautham B A
ad483fd66e
HDFS-15740. Add x-platform utilities (#2567) 2021-01-29 10:11:38 -08:00
Jim Brennan
f565476939 [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 2021-01-28 18:17:15 +00:00
Jim Brennan
6184ab07f6 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 2021-01-28 16:47:52 +00:00
sunlisheng
f8769e0f4b HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 2021-01-28 10:10:39 +08:00
Ayush Saxena
e40f99f6d5 HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun. 2021-01-26 14:25:27 +05:30
Akira Ajisaka
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 2021-01-21 10:07:46 +09:00
He Xiaoqiao
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Masatake Iwasaki
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-14 16:19:35 +09:00
touchida
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-13 11:23:07 +09:00
Yiqun Lin
0d7ac54510 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 2021-01-12 23:38:55 +08:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
He Xiaoqiao
ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
Ayush Saxena
2ba7ec2b48 HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun. 2021-01-08 16:27:54 +05:30
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang
2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 2021-01-04 20:54:44 -08:00
Ayush Saxena
77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 2021-01-04 09:43:58 -08:00
Masatake Iwasaki
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. (#2501) 2021-01-04 14:00:57 +09:00
Ayush Saxena
1cd96e8dd8 HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. 2021-01-01 11:37:52 +05:30
maobaolong
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable (#2572) 2020-12-28 14:18:18 +08:00
hemanthboyina
16a20503ca HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 2020-12-22 17:27:31 +05:30
langlaile1221
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566) 2020-12-22 09:25:11 +08:00
Ayush Saxena
ba496f61f8 HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina. 2020-12-22 02:16:00 +05:30
Ayush Saxena
5ab1db9e21 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 2020-12-21 09:58:30 +05:30
Ayush Saxena
2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ayush Saxena
3ecd3628d4 HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang. 2020-12-20 15:18:25 +05:30
Ayush Saxena
70789fc6b9 HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao. 2020-12-20 15:12:32 +05:30
Ahmed Hussein
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-17 11:33:29 -06:00
Ayush Saxena
0da00fc743 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-14 20:42:23 -08:00
liuyanpunk
3363cc380b HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534) 2020-12-15 10:06:11 +05:30
S O'Donnell
9ed737001c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-11 20:53:10 +00:00
Ahmed Hussein
be35fa186c
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 11:02:42 -06:00
Neil
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config (#2532) 2020-12-10 13:47:00 -08:00
Ahmed Hussein
3ec01b1bb3
HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 13:09:53 -06:00
Ahmed Hussein
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 2020-12-08 13:25:24 -06:00
Gautham B A
4ffec79b7c
HDFS-15712. Upgrade googletest to 1.10.0 (#2523) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-12-08 17:08:14 +09:00
crossfire
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 2020-12-07 15:49:45 -08:00
Ahmed Hussein
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. (#2516) 2020-12-07 14:22:06 -06:00
Yiqun Lin
da1ea2530f HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 2020-12-07 18:52:12 +08:00
Ayush Saxena
ad40715690 HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun. 2020-12-07 15:00:19 +05:30
Ayush Saxena
7dda804a1a HDFS-14090. RBF: Improved isolation for downstream name nodes. {Static}. Contributed by Fengnan Li and CR Hota. 2020-12-04 22:56:37 +05:30
Hui Fei
e2c1268ebd HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 09:20:09 +08:00
Ahmed Hussein
07655a7040
HDFS-15706. HttpFS: Log more information on request failures. (#2515) 2020-12-03 15:14:40 -08:00
Wei-Chiu Chuang
42a29199c0 HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma. 2020-12-02 21:32:02 -08:00
LeonGao
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483). Contributed by Leon Gao. 2020-12-02 15:53:09 -08:00
Ahmed Hussein
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 15:38:20 -06:00
Ahmed Hussein
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 13:59:00 -06:00
Ahmed Hussein
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 2020-12-01 15:02:53 -06:00
Masatake Iwasaki
6a1d7d9ed2
HDFS-15677. TestRouterRpcMultiDestination#testGetCachedDatanodeReport fails on trunk. (#2503) 2020-12-01 08:06:52 +09:00
Hui Fei
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495) 2020-11-28 12:19:57 +08:00
Akira Ajisaka
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:32:10 +09:00
Akira Ajisaka
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs (#2488)" because JIRA ID is missing. 
This reverts commit ce5b3d7975.
 2020-11-26 11:30:53 +09:00
Akira Ajisaka
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-11-26 11:27:30 +09:00
Siyao Meng
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472) 2020-11-25 11:01:04 -08:00
He Xiaoqiao
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh
d73029463c
HDFS-15690. Add lz4-java as test dependency (#2481) 2020-11-21 17:49:56 -08:00
Ahmed Hussein
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2020-11-20 18:03:45 +05:30
Masatake Iwasaki
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2020-11-18 16:11:09 +09:00
Akira Ajisaka
b57f04cd5b
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 2020-11-17 10:56:56 +09:00
Takanobu Asanuma
f56cd88d7d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 2020-11-13 11:02:26 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
Peter Bacsko
cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak." 
This reverts commit b0ab222a6c.
 2020-11-11 17:06:12 +01:00
Peter Bacsko
b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
Siyao Meng
fbd2220167
HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir (#2448) 2020-11-10 12:13:15 -08:00
Yizhong Zhang
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 2020-11-10 09:11:31 -08:00
Hui Fei
f1d5652359 HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang. 2020-11-10 13:09:03 +08:00
maobaolong
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 2020-11-10 13:01:10 +08:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
He Xiaoqiao
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
 2020-11-04 13:53:46 +08:00
Ayush Saxena
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:28:44 +05:30
Konstantin V Shvachko
d07dc7afb4 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 2020-11-03 12:02:40 -08:00
Ahmed Hussein
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2020-10-28 16:24:34 -07:00
Akira Ajisaka
b3ba74d72d
HDFS-15657. RBF: TestRouter#testNamenodeHeartBeatEnableDefault fails by BindException (#2418) 2020-10-28 19:12:23 +09:00
Ahmed Hussein
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406) 2020-10-27 18:56:40 -07:00
Ahmed Hussein
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2020-10-27 18:52:56 -07:00
Hui Fei
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 2020-10-28 09:13:25 +08:00
Ahmed Hussein
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 20:45:00 +09:00
Akira Ajisaka
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309) 2020-10-27 19:18:08 +09:00
Ahmed Hussein
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 16:56:30 +09:00
Hui Fei
12c908c827 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang
74634eb002 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 2020-10-23 12:05:53 -07:00
Inigo Goiri
e7aa4da0a2 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 2020-10-23 09:47:10 -07:00
Hui Fei
264c948e60 HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang 2020-10-23 14:06:22 +08:00
Kihwal Lee
da1b6e3cc2 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 2020-10-22 16:59:49 -05:00
Kihwal Lee
88a9f42f32 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-21 00:59:35 -05:00
Takanobu Asanuma
30f06e0c74
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 2020-10-20 19:08:59 +09:00
Ayush Saxena
2e8cafac3b HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 2020-10-19 10:48:47 +05:30
Ayush Saxena
173310e2f5 HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang. 2020-10-19 10:34:17 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Kihwal Lee
740a2c4635 HDFS-15627. Log delete audits before collecting blocks. 2020-10-16 11:13:41 -05:00
Ayush Saxena
cc57eebe45
HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2020-10-16 10:36:24 +05:30
Kihwal Lee
e45407128d HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 2020-10-14 17:26:04 -05:00
Siyao Meng
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370) 2020-10-13 10:59:42 -07:00
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same 
Contributed by Gabor Bota.
 2020-10-13 16:17:44 +01:00
Mukund Thakur
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:31:00 +01:00
Steve Loughran
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380)" 
(reverting to change the JIRA number)

This reverts commit 3e1b1dfd77.

Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd
 2020-10-13 13:30:38 +01:00
Mukund Thakur
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.
 2020-10-13 13:27:55 +01:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378). Contributed by Uma Maheswara Rao G. 2020-10-12 14:22:44 -07:00
Akira Ajisaka
69ef9b1ee8
HDFS-15620. RBF: Fix test failures after HADOOP-17281 (#2375) 2020-10-12 09:52:12 +09:00
LeonGao91
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 2020-10-10 22:08:36 +05:30
Hui Fei
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 2020-10-09 16:12:22 +08:00
Karthik Palanisamy
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 2020-10-08 12:51:04 +05:30
Karthik Palanisamy
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 2020-10-07 22:09:04 +05:30
bshashikant
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. (#2355) 2020-10-07 10:34:32 +05:30
Akira Ajisaka
074f0d46af
HDFS-15613. RBF: Router FSCK fails after HDFS-14442. (#2360) 
* Support getHAServiceState in DFSRouter
 2020-10-07 13:37:45 +09:00
hemanthboyina
921ca1f554 HDFS-15543. RBF: Write Should allow, when a subcluster is unavailable for RANDOM mount points with fault Tolerance enabled. Contributed by Hemanth Boyina. 2020-10-07 09:58:53 +05:30
Doroszlai, Attila
1cfe5916e2 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-06 14:58:42 -07:00
Fei Hui
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356). Contributed by Hui Fei. 2020-10-05 22:11:24 +05:30
Siyao Meng
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir (#2352) 2020-10-05 05:02:00 -07:00
huangtianhua
a89ca56a1b
HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347) 2020-09-29 19:00:55 -07:00
Ayush Saxena
d686a655bc HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui. 2020-09-29 00:03:11 +05:30
Ayush Saxena
fc8a6dd8a9 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 2020-09-28 23:55:31 +05:30
Fei Hui
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345) 2020-09-28 09:54:01 -07:00
Inigo Goiri
eae0035a2d HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao. 2020-09-28 09:40:39 -07:00
Takanobu Asanuma
d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Uma Maheswara Rao G
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00
Ye Ni
00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332) 2020-09-25 09:47:54 -07:00
Uma Maheswara Rao G
3ccc962b99
HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-24 07:07:48 -07:00
Aryan Gupta
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-09-24 21:15:55 +09:00
huangtianhua
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
bshashikant
368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326) 2020-09-24 14:00:41 +05:30
Ye Ni
ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted (#2274) 2020-09-22 13:23:20 -07:00
Hui Fei
474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
Kihwal Lee
dfc2682213 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 2020-09-22 10:53:04 -05:00
lfengnan
3e8b1e7426
HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266). Contributed by Fengnan Li. 2020-09-21 23:24:13 +05:30
Ayush Saxena
7a6265ac42 HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 2020-09-21 08:50:08 +05:30
Ayush Saxena
8eba922bb3 HDFS-15579. RBF: The constructor of PathLocation may got some misunderstanding. Contributed by Janus Chow. 2020-09-20 08:54:25 +05:30
Ayush Saxena
2377df38ad HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:18:10 +05:30
S O'Donnell
20a0e6278d HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-17 14:11:42 +01:00
bshashikant
425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296) 2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-16 22:43:00 -07:00
Takanobu Asanuma
5c5b2ed7c7 HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui. 2020-09-16 01:36:46 +09:00
hemanthboyina
f4ed9f3f91 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-15 10:10:21 +05:30
He Xiaoqiao
924959088e HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun. 2020-09-15 12:05:04 +08:00
Siyao Meng
6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258) 2020-09-14 13:31:34 -07:00
Akira Ajisaka
c78d18023d
HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267) 2020-09-14 11:34:52 +09:00
Uma Maheswara Rao G
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298). Contributed by Uma Maheswara Rao G. 2020-09-12 17:06:39 -07:00
Mingliang Liu
f59f7f2175
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell 2020-09-12 01:41:38 -07:00
hemanthboyina
2a6fdcedeb HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui. 2020-09-11 18:21:25 +05:30
imbajin
89428f142f
HDFS-15551. Tiny Improve for DeadNode detector (#2265) 
Contributed by imbajin.

Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-09-11 15:44:03 +08:00
1996fanrui
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache (#2241) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-09-10 22:30:52 -07:00
zz
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree (#2185). Contributed by Zhenzhao Wang. 
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
 2020-09-10 21:20:32 -07:00
Siyao Meng
43c52d6495
HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir (#2295) 
Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
 2020-09-10 15:55:23 +05:30
He Xiaoqiao
1d6d0d8207 HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun. 
Co-authored-by: Chen Zhang <chzhang1987@gmail.com>

Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-09-09 20:09:13 +08:00
Uma Maheswara Rao G
ac7d4623ae
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275) Contributed by Uma Maheswara Rao G. 2020-09-07 11:36:13 -07:00
Uma Maheswara Rao G
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well (#2234). Contributed by Uma Maheswara Rao G. 2020-09-03 11:06:20 -07:00
He Xiaoqiao
60de592a88 HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree. 2020-08-31 14:49:12 +08:00
bshashikant
2d03209a06
HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion (#2251) 2020-08-31 11:29:48 +05:30
Steve Loughran
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing (#2252) 
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
 2020-08-28 15:39:09 +01:00
Vivek Ratnavel Subramanian
06793da100
HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory (#2230) 2020-08-27 11:36:30 -07:00
Tsz-Wo Nicholas Sze
41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. (#2233) 2020-08-27 14:54:52 +05:30
Stephen O'Donnell
2ffe00fc46 HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-08-26 23:05:32 -07:00
Takanobu Asanuma
ca8e7a7725 HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina. 2020-08-27 12:10:39 +09:00
hemanthboyina
6e618b6a7e HDFS-15536. RBF: Clear Quota in Router was not consistent. 2020-08-26 13:03:08 +05:30
S O'Donnell
6a49bf9bff HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 2020-08-25 15:18:36 +01:00
He Xiaoqiao
a7830423c5 HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu. 2020-08-24 21:16:00 +08:00
hemanthboyina
83fd15b412 HDFS-14504. Rename with Snapshots does not honor quota limit. 2020-08-23 16:10:47 +05:30
Chen Liang
f734455e5d HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 15:25:10 -07:00
Ayush Saxena
4813a37023 HDFS-15535. RBF: Fix Namespace path to snapshot path resolution for snapshot API. Contributed by Ayush Saxena. 2020-08-20 23:54:35 +05:30
Uma Maheswara Rao G
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 2020-08-19 09:30:41 -07:00
Siyao Meng
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS (#2220) 2020-08-18 19:28:19 +09:00
bshashikant
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. (#2172) 
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.

* Addressed review comments.

* Rebased to latest trunk and added snapshotTrashRoot config check.

* Addressed review comments.

* Removede empty line added in SnapshotManager.Java.

* Addressed whitespace issues.
 2020-08-17 12:56:13 +05:30
Ayush Saxena
e3d1966f58 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 2020-08-15 15:07:58 +05:30
Tsz-Wo Nicholas Sze
b93dd7c281
HDFS-15519. Check inaccessible INodes in FsImageValidation. (#2224) 2020-08-14 10:10:01 -07:00
bshashikant
15a76e8d65
HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. (#2219) 2020-08-14 20:46:48 +05:30
Vivek Ratnavel Subramanian
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots (#2212) 2020-08-13 22:36:15 +05:30
Aryan Gupta
e592ec5f8b
HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. (#2217) 
* HDFS-15518. Fixed String operationName = ListSnapshot.

* HDFS-15518. Changed ListSnapshot to operationName.

Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>
 2020-08-13 00:09:38 +05:30
S O'Donnell
10716040a8 HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang 2020-08-12 09:02:47 +01:00
Tsz-Wo Nicholas Sze
11cec9ab94
HDFS-15523. Fix findbugs warnings from HDFS-15520. (#2218) 2020-08-11 21:41:28 -07:00
Siyao Meng
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory (#2176) 2020-08-11 08:52:16 -07:00
Gautham B A
909f1e82d3
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-11 16:35:08 +09:00
Uma Maheswara Rao G
8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test
 2020-08-11 00:01:58 -07:00
Tsz-Wo Nicholas Sze
592127bdc2
HDFS-15520 Use visitor pattern to visit namespace tree (#2203) 2020-08-11 12:30:30 +05:30
Akira Ajisaka
32895f4f7e
HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 2020-08-11 13:49:56 +09:00
Akira Ajisaka
7938ebfb9d
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 2020-08-11 11:14:58 +09:00
Akira Ajisaka
9062814bab
HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module 2020-08-11 10:58:56 +09:00
Ayush Saxena
c2a17659d1 HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team. 2020-08-08 14:33:53 +05:30
Ayush Saxena
975b6024dd HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:19:17 +05:30
Takanobu Asanuma
1d5ccc790b
HDFS-15512. Remove smallBufferSize in DFSClient. (#2191) 2020-08-06 22:47:12 +05:30
Mingliang Liu
58def7cecb
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 20:48:45 -07:00
bshashikant
e072d33327
HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable (#2175) 2020-08-04 14:10:29 +05:30
hemanthboyina
82f3ffcd64 HDFS-15503. File and directory permissions are not able to be modified from WebUI. 2020-08-03 23:21:01 +05:30
Akira Ajisaka
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors (#2098) 2020-08-03 10:46:51 +09:00
bshashikant
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. (#2181) 2020-08-02 21:59:56 +05:30
hemanthboyina
528a799a78 HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree. 2020-08-02 12:02:37 +05:30
Yuan
e756fe3590
HDFS-14950. fix missing libhdfspp lib in dist-package (#1947) 
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
 2020-07-31 16:49:49 +09:00
Tsz-Wo Nicholas Sze
05b3337a46
HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots (#2165) 2020-07-30 23:06:51 +05:30
bshashikant
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. (#2166) 2020-07-29 21:33:25 +05:30
touchida
026dce5334
HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. (#2135) 2020-07-27 09:55:11 -07:00
bshashikant
2d12496643
HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr (#2163) 2020-07-22 16:16:27 -07:00
Uma Maheswara Rao G
ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 2020-07-21 23:29:10 -07:00
Masatake Iwasaki
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Inigo Goiri
8b7695bb26 HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 2020-07-21 09:00:07 -07:00
Tsz-Wo Nicholas Sze
d57462f2da
HDFS-15479. Ordered snapshot deletion: make it a configurable feature (#2156) 2020-07-21 11:36:24 +05:30
Shashikant Banerjee
d9441f95c3 HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee. 2020-07-21 11:13:05 +05:30
Chen Liang
3833c616e0 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 2020-07-20 12:49:58 -07:00
Ayush Saxena
6cbd8854ee HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi. 2020-07-20 21:49:05 +05:30
Tsz-Wo Nicholas Sze
2cec50cf16
HDFS-15463. Add a tool to validate FsImage. (#2140) 2020-07-20 11:44:30 +05:30
Suraj Naik
4101b0c0ed
HDFS-15476 Make AsyncStream executor private (#2151) 2020-07-19 23:26:54 +05:30
Ayush Saxena
8a9a674ef1 HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu. 2020-07-18 15:36:55 +05:30
Ayush Saxena
6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Ahmed Hussein
1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 11:39:06 -05:00
Ahmed Hussein
98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 09:53:18 -05:00
Shashikant Banerjee
85d4718ed7 HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee. 2020-07-15 10:32:25 +05:30
Gautham B A
cce5a6f609
HDFS-15385 Upgrade boost library to 1.72 (#2051) 
* Removed the asio-1.10.2 that was
  checked in under libhdfspp/third_party
  directory.
 2020-07-15 13:39:48 +09:00
zhaoyim
bdce75d737
HDFS-15371. Nonstandard characters exist in NameNode.java (#2032) 
Contributed by zhaoyim
 2020-07-14 14:42:12 +01:00
Steve Loughran
b9fa5e0182
HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:30:02 +01:00
He Xiaoqiao
b97fea65e7 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:12:48 +08:00
lfengnan
84b74b335c
HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110) 2020-07-12 00:10:12 -07:00
Uma Maheswara Rao G
3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 2020-07-11 23:50:04 -07:00
Xiaoyu Yao
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
He Xiaoqiao
4f26454a7d HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang. 2020-07-08 00:02:39 +08:00
Ye Ni
f77bbc2123
HDFS-15312. Apply umask when creating directory by WebHDFS (#2096) 2020-07-06 19:01:46 -07:00
Uma Maheswara Rao G
dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 2020-07-06 18:50:03 -07:00
Ye Ni
e820baa6e6
HDFS-15417. RBF: Get the datanode report from cache for federation WebHDFS operations (#2080) 2020-07-06 16:17:09 -07:00
Shanyu Zhao
834372f404
HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-06 23:43:34 +08:00
Uma Maheswara Rao G
55a2ae80dc
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 2020-07-04 13:45:49 -07:00
Ayush Saxena
f86f15cf20 HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 13:00:43 +05:30
Uma Maheswara Rao G
1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-07-04 00:12:10 -07:00
He Xiaoqiao
9ac498e300 HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu. 2020-07-01 12:38:07 +08:00
Stephen O'Donnell
2a67e2b1a0 HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-30 07:10:17 -07:00
Akira Ajisaka
c71ce7ac33
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 2020-06-28 16:02:47 +09:00
Ayush Saxena
8db38c98a6 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 2020-06-27 19:13:01 +05:30
Virajith Jalaparti
bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle
 2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G
d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-06-26 01:29:38 -07:00
lfengnan
84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch (#2047) 
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
 2020-06-23 13:12:29 -07:00
Uma Maheswara Rao G
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 2020-06-23 01:42:25 -07:00
Ayush Saxena
17ffcab5f6 HDFS-14546. Document block placement policies. Contributed by Amithsha. 2020-06-21 17:27:28 +05:30
Uma Maheswara Rao G
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 2020-06-20 00:32:02 -07:00
Stephen O'Donnell
d50e93ce7b HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang
edf716a5c3 Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." 
This reverts commit 730a39d138.
 2020-06-18 06:45:15 -07:00
S O'Donnell
123777823e HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 2020-06-18 12:26:22 +01:00
Yiqun Lin
9cbd76cc77 HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
Ayush Saxena
c8ed33cd2a HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor. 2020-06-17 13:55:40 +05:30
Stephen O'Donnell
730a39d138 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-16 15:59:13 -07:00
Eric Yang
81d8a887b0
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-06-15 10:56:16 +09:00
Takanobu Asanuma
f41a144077 HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 2020-06-15 09:15:53 +09:00
Inigo Goiri
719b53a79d HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 2020-06-13 09:35:05 -07:00
Uma Maheswara Rao G
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 2020-06-12 14:32:19 -07:00
Vinayakumar B
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G
93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 2020-06-10 15:00:02 -07:00
Ayush Saxena
b735a77717 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:06:16 +05:30
Ayush Saxena
635e6a16d0 HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi. 2020-06-10 11:50:37 +05:30
Ayush Saxena
8525874561 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:53:33 +05:30
Uma Maheswara Rao G
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 2020-06-06 08:11:57 -07:00
Ayush Saxena
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 2020-06-05 22:19:38 -07:00
Uma Maheswara Rao G
76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 2020-06-05 10:58:21 -07:00
Ayush Saxena
2326123705 HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena. 2020-06-05 19:15:05 +05:30
Toshihiro Suzuki
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.
 2020-06-05 11:11:49 +01:00
Ayush Saxena
e8cb2ae409 HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 2020-06-03 12:47:15 +05:30
Ayush Saxena
f6453244ab HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan. 2020-06-03 12:16:36 +05:30
Uma Maheswara Rao G
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00
Ayush Saxena
ae13a5ccbe HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 2020-05-31 17:10:59 +05:30
Wanqiang Ji
d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Ayush Saxena
a838d871a7 HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2020-05-28 20:52:58 +05:30
Ayush Saxena
9b38be43c6 HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:00:04 +05:30
Inigo Goiri
2148a8fe64 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 2020-05-27 11:06:13 -07:00
Ayush Saxena
6c9f75cf16 HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena. 2020-05-26 16:26:57 +05:30
Ayush Saxena
f43a152b97 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:03:41 +05:30
Ayush Saxena
f4901d0778 HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun. 2020-05-24 17:59:16 +05:30
Inigo Goiri
6e04b00df1 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 2020-05-23 09:58:19 -07:00
Ayush Saxena
e0ae232f66 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 2020-05-23 18:38:32 +05:30
Ayush Saxena
f99fcb26ab HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2020-05-23 18:29:31 +05:30
Takanobu Asanuma
4d22d1c58f HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 2020-05-23 17:28:38 +09:00
Uma Maheswara Rao G
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 2020-05-21 21:34:58 -07:00
leosunli
be374faf42
HDFS-13639. SlotReleaser is not fast enough (#1885) 2020-05-21 13:21:17 -07:00
Yiqun Lin
1983eea62d HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun. 2020-05-20 10:39:40 +08:00
Abhishek Das
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 2020-05-18 22:27:12 -07:00
Chen Liang
7bb902bc0d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 10:58:52 -07:00
Ayush Saxena
c84e6beada HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 2020-05-18 22:24:34 +05:30
Wei-Chiu Chuang
2abcf7762a HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang
4525292d41 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit 86e6aa8eec.
 2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang
50caba1a92 HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 2020-05-18 08:40:38 -07:00