big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,482 Commits 2 Branches 2 Tags 264 MiB
7a8c2e0097
Commit Graph

8645 Commits

Author SHA1 Message Date
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani
 2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
Ahmed Hussein
352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097) 2021-06-11 11:00:42 -05:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-11 17:30:55 +09:00
Viraj Jasani
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
Ahmed Hussein
a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058) 2021-06-08 11:09:31 -05:00
litao
c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena
bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Konstantin V Shvachko
8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko
1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Hui Fei
98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Narges Shadab
3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena
fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Siyao Meng
8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958) 2021-05-10 12:41:07 -07:00
Ayush Saxena
359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
zhuqi
76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma
01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu
c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
Viraj Jasani
acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao
85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Vivek Ratnavel Subramanian
c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Viraj Jasani
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Stephen O'Donnell
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
He Xiaoqiao
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
zhuqi
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Brahma Reddy Battula
ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula
0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
He Xiaoqiao
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Viraj Jasani
90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:22:54 +05:30
Ayush Saxena
cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
He Xiaoqiao
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:24:14 +08:00
touchida
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Viraj Jasani
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:47:45 -07:00
Ayush Saxena
569e407f64 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:10:02 +05:30
Jack Jiang
9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof (#2805) 2021-03-23 08:51:35 -07:00
Narges Shadab
d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 17:06:58 +05:30
litao
343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 2021-03-22 20:29:17 +09:00
He Xiaoqiao
03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-22 18:37:00 +08:00
Viraj Jasani
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-19 20:37:44 -07:00
Narges Shadab
f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-19 16:36:48 +08:00
Viraj Jasani
9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 2021-03-17 15:12:36 +09:00
Takanobu Asanuma
9ba60c33d8 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 2021-03-17 14:14:26 +09:00
touchida
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-17 12:00:05 +09:00
Hui Fei
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 2021-03-17 09:11:30 +08:00
sunlisheng
7025f39944 HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 2021-03-15 11:34:13 +08:00
He Xiaoqiao
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun. 2021-03-14 17:48:18 +08:00
lamberken
6921ec8b40
HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751). Contributed by Xie Lei. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-13 14:22:11 +08:00
Xiaoyu Yao
7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764) 2021-03-12 11:27:53 -08:00
Takanobu Asanuma
ecd3335187 HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel. 2021-03-12 15:42:15 +09:00
Hui Fei
6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate (#2746) 2021-03-10 14:11:29 +08:00
litao
6a922f9896
HDFS-15873. Add namenode address in logs for block report (#2743) 2021-03-08 11:50:51 -05:00
tomscut
9cb51bf106 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 2021-03-06 13:40:44 -08:00
litao
cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
 2021-03-03 22:34:02 +09:00
zhuqi
1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. (#2721). Contributed by Qi Zhu 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-02 13:16:11 +08:00
Konstantin V Shvachko
9501c698f4 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 2021-03-01 17:17:23 -08:00
litao
32353eb38a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718) 
Authored-by: tomscut <litao@bigo.sg>
 2021-03-01 07:52:59 -08:00
S O'Donnell
c3b3b36dee HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 2021-03-01 11:36:41 +00:00
S O'Donnell
9c583f36d5 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 2021-02-24 10:29:36 +00:00
LeonGao
940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704) 2021-02-23 22:29:48 -08:00
Ayush Saxena
647d23b611 HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:08:38 +05:30
Takanobu Asanuma
a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714) 2021-02-23 15:12:06 +09:00
Wang Yuxuan
1e32ab7d2f HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang
f8059d0bdf Revert "first commit (#2559)" 
This reverts commit 1b94569d89.
 2021-02-22 18:13:06 +08:00
Wang Yuxuan
1b94569d89
first commit (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 02:10:39 -08:00
jianghuazhu
98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-02-21 23:22:32 -08:00
Ayush Saxena
ff84a57483 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 2021-02-20 19:48:00 +05:30
Takanobu Asanuma
25af8901a9 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 2021-02-19 14:44:58 +09:00
Hui Fei
2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:07:22 +08:00
Ayush Saxena
e391844e8e HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. 2021-02-18 01:39:04 +05:30
Akira Ajisaka
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Zehao Chen
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
litao
c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676) 2021-02-14 00:12:24 -05:00
Steve Loughran
798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
LeonGao
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 2021-02-09 10:49:28 +00:00
bshashikant
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. (#2677) 2021-02-09 10:27:51 +05:30
LeonGao
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625) 2021-02-08 11:01:21 -08:00
bshashikant
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682) 2021-02-06 08:26:12 -08:00
Jim Brennan
c4918fb298 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 21:20:30 +00:00
fermi
7ef2875c89 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 2021-02-05 11:01:49 +08:00
maobaolong
b99dba38b9
HDFS-15819. Fix import style #2681 2021-02-05 10:11:27 +08:00
Hui Fei
15a1f7adfc HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 2021-02-04 09:19:51 +08:00
Kihwal Lee
182623d2bc HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 2021-02-03 08:55:50 -06:00
Hui Fei
66ecee333e HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang 2021-02-03 20:00:45 +08:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
crossfire
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 2021-02-02 09:02:09 +00:00
He Xiaoqiao
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:50:54 +08:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
He Xiaoqiao
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:28:29 +08:00
Narges Shadab
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 2021-02-01 01:56:59 -08:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
Jim Brennan
f565476939 [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 2021-01-28 18:17:15 +00:00
Jim Brennan
6184ab07f6 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 2021-01-28 16:47:52 +00:00
sunlisheng
f8769e0f4b HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 2021-01-28 10:10:39 +08:00
Akira Ajisaka
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 2021-01-21 10:07:46 +09:00
He Xiaoqiao
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Masatake Iwasaki
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-14 16:19:35 +09:00
touchida
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-13 11:23:07 +09:00
Masatake Iwasaki
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
He Xiaoqiao
ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 2021-01-04 09:43:58 -08:00
Masatake Iwasaki
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. (#2501) 2021-01-04 14:00:57 +09:00
Ayush Saxena
1cd96e8dd8 HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. 2021-01-01 11:37:52 +05:30
maobaolong
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable (#2572) 2020-12-28 14:18:18 +08:00
hemanthboyina
16a20503ca HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 2020-12-22 17:27:31 +05:30
langlaile1221
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566) 2020-12-22 09:25:11 +08:00
Ayush Saxena
ba496f61f8 HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina. 2020-12-22 02:16:00 +05:30
Ayush Saxena
2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ayush Saxena
3ecd3628d4 HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang. 2020-12-20 15:18:25 +05:30
Ayush Saxena
70789fc6b9 HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao. 2020-12-20 15:12:32 +05:30
Ahmed Hussein
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-17 11:33:29 -06:00
Ayush Saxena
0da00fc743 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-14 20:42:23 -08:00
liuyanpunk
3363cc380b HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534) 2020-12-15 10:06:11 +05:30
S O'Donnell
9ed737001c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-11 20:53:10 +00:00
Ahmed Hussein
be35fa186c
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 11:02:42 -06:00
Neil
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config (#2532) 2020-12-10 13:47:00 -08:00
Ahmed Hussein
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 2020-12-08 13:25:24 -06:00
crossfire
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 2020-12-07 15:49:45 -08:00
Ahmed Hussein
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. (#2516) 2020-12-07 14:22:06 -06:00
Ayush Saxena
ad40715690 HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun. 2020-12-07 15:00:19 +05:30
Hui Fei
e2c1268ebd HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 09:20:09 +08:00
Wei-Chiu Chuang
42a29199c0 HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma. 2020-12-02 21:32:02 -08:00
LeonGao
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483). Contributed by Leon Gao. 2020-12-02 15:53:09 -08:00
Ahmed Hussein
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 15:38:20 -06:00
Ahmed Hussein
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 13:59:00 -06:00
Ahmed Hussein
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 2020-12-01 15:02:53 -06:00
Hui Fei
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495) 2020-11-28 12:19:57 +08:00
Siyao Meng
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check (#2472) 2020-11-25 11:01:04 -08:00
He Xiaoqiao
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2020-11-23 11:26:52 +08:00
Liang-Chi Hsieh
d73029463c
HDFS-15690. Add lz4-java as test dependency (#2481) 2020-11-21 17:49:56 -08:00
Ahmed Hussein
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2020-11-20 18:03:45 +05:30
Masatake Iwasaki
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. (#2467) 2020-11-18 16:11:09 +09:00
Takanobu Asanuma
f56cd88d7d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 2020-11-13 11:02:26 +09:00
Issac Buenrostro
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… (#2255) 
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
 2020-11-12 14:57:26 -08:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
Yizhong Zhang
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 2020-11-10 09:11:31 -08:00
maobaolong
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 2020-11-10 13:01:10 +08:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
He Xiaoqiao
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
 2020-11-04 13:53:46 +08:00
Ayush Saxena
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:28:44 +05:30
Konstantin V Shvachko
d07dc7afb4 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 2020-11-03 12:02:40 -08:00
Ahmed Hussein
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently (#2419) 2020-10-28 16:24:34 -07:00
Ahmed Hussein
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. (#2406) 2020-10-27 18:56:40 -07:00
Ahmed Hussein
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2020-10-27 18:52:56 -07:00
Hui Fei
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 2020-10-28 09:13:25 +08:00
Ahmed Hussein
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 20:45:00 +09:00
Akira Ajisaka
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309) 2020-10-27 19:18:08 +09:00
Ahmed Hussein
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
 2020-10-27 16:56:30 +09:00
Hui Fei
12c908c827 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang
74634eb002 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 2020-10-23 12:05:53 -07:00
Inigo Goiri
e7aa4da0a2 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 2020-10-23 09:47:10 -07:00
Kihwal Lee
da1b6e3cc2 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 2020-10-22 16:59:49 -05:00
Kihwal Lee
88a9f42f32 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-21 00:59:35 -05:00
Ayush Saxena
2e8cafac3b HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 2020-10-19 10:48:47 +05:30
Ayush Saxena
173310e2f5 HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang. 2020-10-19 10:34:17 +05:30
Ayush Saxena
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Kihwal Lee
740a2c4635 HDFS-15627. Log delete audits before collecting blocks. 2020-10-16 11:13:41 -05:00
Siyao Meng
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled (#2370) 2020-10-13 10:59:42 -07:00
Gabor Bota
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same 
Contributed by Gabor Bota.
 2020-10-13 16:17:44 +01:00
Mukund Thakur
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.

Change-Id: I6cec41c20467920cf21f169653553535414b2680
 2020-10-13 13:31:00 +01:00
Steve Loughran
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380)" 
(reverting to change the JIRA number)

This reverts commit 3e1b1dfd77.

Change-Id: I8261357f38268b5d7c1c13f6ef366d037be972dd
 2020-10-13 13:30:38 +01:00
Mukund Thakur
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing (#2380) 
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.

Contributed by Mukund Thakur.
 2020-10-13 13:27:55 +01:00
Konstantin V Shvachko
b3786d6c3c HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:26:49 -07:00
Uma Maheswara Rao G
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup (#2378). Contributed by Uma Maheswara Rao G. 2020-10-12 14:22:44 -07:00
LeonGao91
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 2020-10-10 22:08:36 +05:30
Karthik Palanisamy
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 (#2365) 2020-10-08 12:51:04 +05:30
Karthik Palanisamy
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 2020-10-07 22:09:04 +05:30
bshashikant
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. (#2355) 2020-10-07 10:34:32 +05:30
Fei Hui
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently (#2356). Contributed by Hui Fei. 2020-10-05 22:11:24 +05:30
Siyao Meng
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir (#2352) 2020-10-05 05:02:00 -07:00
Fei Hui
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345) 2020-09-28 09:54:01 -07:00
Takanobu Asanuma
d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Vinayakumar B
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30