big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
25,037 Commits 2 Branches 2 Tags 264 MiB
fdef2b4cca
Commit Graph

6068 Commits

Author SHA1 Message Date
litao
fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Viraj Jasani
d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 2021-06-22 20:21:34 -07:00
Viraj Jasani
c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
litao
bdc9c8809e
HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:11 +05:30
litao
c491f81a30
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:20:52 +05:30
LeonGao
643dfd60e4
HDFS-15842. HDFS mover to emit metrics. (#2738) 2021-06-19 15:39:46 -07:00
hust_hhb
56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065) 2021-06-17 11:08:29 +08:00
Viraj Jasani
c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani
 2021-06-14 14:23:50 -07:00
Siyao Meng
ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
litao
51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-11 17:30:55 +09:00
Viraj Jasani
eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
Ahmed Hussein
a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058) 2021-06-08 11:09:31 -05:00
litao
c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
huhaiyang
b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena
b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena
bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Konstantin V Shvachko
1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Narges Shadab
3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
Viraj Jasani
028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
huhaiyang
acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Viraj Jasani
b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Siyao Meng
8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958) 2021-05-10 12:41:07 -07:00
Ayush Saxena
359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
zhuqi
76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani
041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Wei-Chiu Chuang
b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
bshashikant
ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell
605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao
b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma
01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula
2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang
90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu
c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
litao
85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao
af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
zhangshuyan0
14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Vivek Ratnavel Subramanian
c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Stephen O'Donnell
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
He Xiaoqiao
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Viraj Jasani
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
zhuqi
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Brahma Reddy Battula
ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
He Xiaoqiao
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Ayush Saxena
cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
He Xiaoqiao
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:24:14 +08:00