big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24,799 Commits 2 Branches 2 Tags
Commit Graph

6028 Commits

Author SHA1 Message Date
Kihwal Lee
182623d2bc HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 2021-02-03 08:55:50 -06:00
Hui Fei
66ecee333e HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang 2021-02-03 20:00:45 +08:00
YaYun-Wang
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
crossfire
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 2021-02-02 09:02:09 +00:00
He Xiaoqiao
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:28:29 +08:00
Narges Shadab
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 2021-02-01 01:56:59 -08:00
Siyao Meng
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
Jim Brennan
f565476939 [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 2021-01-28 18:17:15 +00:00
Akira Ajisaka
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 2021-01-21 10:07:46 +09:00
LeonGao
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
lfengnan
87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 2021-01-04 09:43:58 -08:00
Ayush Saxena
1cd96e8dd8 HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. 2021-01-01 11:37:52 +05:30
maobaolong
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable (#2572) 2020-12-28 14:18:18 +08:00
hemanthboyina
16a20503ca HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 2020-12-22 17:27:31 +05:30
langlaile1221
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566) 2020-12-22 09:25:11 +08:00
Ayush Saxena
2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ayush Saxena
3ecd3628d4 HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang. 2020-12-20 15:18:25 +05:30
Ayush Saxena
70789fc6b9 HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao. 2020-12-20 15:12:32 +05:30
Ahmed Hussein
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-17 11:33:29 -06:00
Ayush Saxena
0da00fc743 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-14 20:42:23 -08:00
liuyanpunk
3363cc380b HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534) 2020-12-15 10:06:11 +05:30
S O'Donnell
9ed737001c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-11 20:53:10 +00:00
Ahmed Hussein
be35fa186c
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 11:02:42 -06:00
Neil
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config (#2532) 2020-12-10 13:47:00 -08:00
crossfire
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 2020-12-07 15:49:45 -08:00
Ahmed Hussein
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. (#2516) 2020-12-07 14:22:06 -06:00
Ayush Saxena
ad40715690 HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun. 2020-12-07 15:00:19 +05:30
Hui Fei
e2c1268ebd HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 09:20:09 +08:00
Wei-Chiu Chuang
42a29199c0 HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma. 2020-12-02 21:32:02 -08:00
LeonGao
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration (#2483). Contributed by Leon Gao. 2020-12-02 15:53:09 -08:00
Ahmed Hussein
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 15:38:20 -06:00
Ahmed Hussein
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 13:59:00 -06:00
Ahmed Hussein
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 2020-12-01 15:02:53 -06:00
Hui Fei
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495) 2020-11-28 12:19:57 +08:00
Takanobu Asanuma
f56cd88d7d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 2020-11-13 11:02:26 +09:00
jianghuazhu
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. (#2138) 2020-11-10 16:34:17 -08:00
Yizhong Zhang
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 2020-11-10 09:11:31 -08:00
maobaolong
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 2020-11-10 13:01:10 +08:00
LeonGao
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount (#2288). Contributed by Leon Gao. 2020-11-09 15:06:16 -08:00
He Xiaoqiao
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
 2020-11-04 13:53:46 +08:00
Ayush Saxena
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:28:44 +05:30
Konstantin V Shvachko
d07dc7afb4 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 2020-11-03 12:02:40 -08:00
Akira Ajisaka
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails (#2309) 2020-10-27 19:18:08 +09:00
Hui Fei
12c908c827 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 2020-10-26 22:12:27 +08:00
Wei-Chiu Chuang
74634eb002 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 2020-10-23 12:05:53 -07:00
Kihwal Lee
da1b6e3cc2 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 2020-10-22 16:59:49 -05:00
Kihwal Lee
88a9f42f32 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2020-10-21 00:59:35 -05:00
Ayush Saxena
2e8cafac3b HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 2020-10-19 10:48:47 +05:30
Ayush Saxena
173310e2f5 HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang. 2020-10-19 10:34:17 +05:30