big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,908 Commits 2 Branches 2 Tags 264 MiB
8f850b46a0
Commit Graph

8474 Commits

Author SHA1 Message Date
Ayush Saxena
359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
Akira Ajisaka
9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
zhuqi
76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma
01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu
c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
Viraj Jasani
acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao
85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Vivek Ratnavel Subramanian
c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Viraj Jasani
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Stephen O'Donnell
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
He Xiaoqiao
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
zhuqi
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Brahma Reddy Battula
ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula
0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
He Xiaoqiao
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Viraj Jasani
90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:22:54 +05:30
Ayush Saxena
cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
He Xiaoqiao
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:24:14 +08:00
touchida
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Viraj Jasani
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:47:45 -07:00
Ayush Saxena
569e407f64 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:10:02 +05:30
Jack Jiang
9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof (#2805) 2021-03-23 08:51:35 -07:00
Narges Shadab
d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 17:06:58 +05:30
litao
343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 2021-03-22 20:29:17 +09:00
He Xiaoqiao
03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-22 18:37:00 +08:00
Viraj Jasani
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-19 20:37:44 -07:00
Narges Shadab
f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-19 16:36:48 +08:00
Viraj Jasani
9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 2021-03-17 15:12:36 +09:00
Takanobu Asanuma
9ba60c33d8 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 2021-03-17 14:14:26 +09:00
touchida
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-17 12:00:05 +09:00
Hui Fei
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 2021-03-17 09:11:30 +08:00
sunlisheng
7025f39944 HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 2021-03-15 11:34:13 +08:00
He Xiaoqiao
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun. 2021-03-14 17:48:18 +08:00