Commit Graph

4437 Commits

Author SHA1 Message Date
Jing Zhao
5059958bf4 HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() when loading editlog. Contributed by Jing Zhao. 2015-05-26 11:59:56 -07:00
Zhe Zhang
1a31f1c303 HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of datastreamer threads. Contributed by Rakesh R. 2015-05-26 11:59:56 -07:00
Zhe Zhang
f0628280c3 HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil. Contributed by Zhe Zhang. 2015-05-26 11:59:56 -07:00
Zhe Zhang
b00c663877 HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream (stateful read). Contributed by Jing Zhao 2015-05-26 11:59:55 -07:00
Jing Zhao
1a8139e6ad Fix merge conflicts. 2015-05-26 11:59:55 -07:00
Jing Zhao
9593776e34 HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open. Contributed by Kai Sasaki. 2015-05-26 11:59:55 -07:00
Vinayakumar B
5eb17e5726 HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated as Idempotent (Contributed by Vinayakumar B) 2015-05-26 11:59:55 -07:00
Vinayakumar B
f9eb95c31d HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY commands from standbynode if any (Contributed by Vinayakumar B) 2015-05-26 11:59:54 -07:00
Zhe Zhang
89d3378578 HDFS-8033. Erasure coding: stateful (non-positional) read from files in striped layout. Contributed by Zhe Zhang. 2015-05-26 11:59:54 -07:00
Zhe Zhang
f5d4a95ef5 HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause block id conflicts. Contributed by Jing Zhao. 2015-05-26 11:59:54 -07:00
Jing Zhao
3f2c6938f1 HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream. Contributed by Yi Liu. 2015-05-26 11:59:54 -07:00
Jing Zhao
eb612b0b70 HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last stripe is at the block group boundary. Contributed by Jing Zhao. 2015-05-26 11:59:53 -07:00
Kai Zheng
fcd54ecce2 HDFS-8136. Client gets and uses EC schema when reads and writes a stripping file. Contributed by Kai Sasaki 2015-05-26 11:59:53 -07:00
Zhe Zhang
e8df2581c3 HDFS-8156. Add/implement necessary APIs even we just have the system default schema. Contributed by Kai Zheng. 2015-05-26 11:59:53 -07:00
Uma Maheswara Rao G
014d8675c5 HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration. (Contributed by Uma Maheswara Rao G) 2015-05-26 11:59:52 -07:00
Zhe Zhang
2f11109bb0 HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema in FileSystemLinkResolver. Contributed by Tsz Wo Nicholas Sze. 2015-05-26 11:59:52 -07:00
Zhe Zhang
e9d85bbf30 HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to create BlockReader. Contributed by Tsz Wo Nicholas Sze. 2015-05-26 11:59:52 -07:00
Tsz-Wo Nicholas Sze
e107886d6f HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error. 2015-05-26 11:59:52 -07:00
Zhe Zhang
922631f04f HDFS-8188. Erasure coding: refactor client-related code to sync with HDFS-8082 and HDFS-8169. Contributed by Zhe Zhang. 2015-05-26 11:59:51 -07:00
Vinayakumar B
dfba46ab57 HDFS-8181. createErasureCodingZone sets retryCache state as false always (Contributed by Uma Maheswara Rao G) 2015-05-26 11:59:51 -07:00
Vinayakumar B
a1c9425265 HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for making it ready for transfer to DN (Contributed by Uma Maheswara Rao G) 2015-05-26 11:59:51 -07:00
Jing Zhao
f6e1160ef1 HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks. Contributed by Jing Zhao. 2015-05-26 11:59:46 -07:00
Jing Zhao
ab76e1fe36 HDFS-7937. Erasure Coding: INodeFile quota computation unit tests. Contributed by Kai Sasaki. 2015-05-26 11:59:13 -07:00
Jing Zhao
cd458c38a0 HDFS-8166. DFSStripedOutputStream should not create empty blocks. Contributed by Jing Zhao. 2015-05-26 11:59:13 -07:00
Zhe Zhang
64f8f0a145 HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. Contributed by Hui Zheng. 2015-05-26 11:59:12 -07:00
Tsz-Wo Nicholas Sze
4c039b0876 HDFS-7994. Detect if resevered EC Block ID is already used during namenode startup. Contributed by Hui Zheng 2015-05-26 11:59:12 -07:00
Jing Zhao
5e8837dd6c HDFS-8120. Erasure coding: created util class to analyze striped block groups. Contributed by Zhe Zhang and Li Bo. 2015-05-26 11:59:12 -07:00
Vinayakumar B
ceb3d1c170 HDFS-7349. Support DFS command for the EC encoding (Contributed by Vinayakumar B) 2015-05-26 11:59:11 -07:00
Vinayakumar B
2c277802c1 HDFS-8027. Erasure Coding: Update CHANGES-HDFS-7285.txt with branch commits (Vinayakumar B) 2015-05-26 11:59:11 -07:00
Vinayakumar B
04c23d95d0 HDFS-8123. Erasure Coding: Better to move EC related proto messages to a separate erasurecoding proto file (Contrubuted by Rakesh R) 2015-05-26 11:59:11 -07:00
Zhe Zhang
3b56a5aa9e HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this operation fails. Contributed by Rakesh R. 2015-05-26 11:59:10 -07:00
Zhe Zhang
a32c4dc38a HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone. Contributed by Vinayakumar B. 2015-05-26 11:59:06 -07:00
Zhe Zhang
35797b0889 HDFS-7936. Erasure coding: resolving conflicts in the branch when merging trunk changes (this commit mainly addresses HDFS-8081 and HDFS-8048. Contributed by Zhe Zhang. 2015-05-26 11:58:40 -07:00
Vinayakumar B
f53e402635 HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all ECSchemas loaded in Namenode. (Contributed by Vinayakumar B) 2015-05-26 11:58:40 -07:00
Kai Zheng
4970f2a2ab HDFS-7889 Subclass DFSOutputStream to support writing striping layout files. Contributed by Li Bo 2015-05-26 11:58:40 -07:00
Jing Zhao
4d0bc724f2 HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks. Contributed by Jing Zhao and Zhe Zhang. 2015-05-26 11:58:32 -07:00
Kai Zheng
c4a0d88366 HDFS-8104 Make hard-coded values consistent with the system default schema first before remove them. Contributed by Kai Zheng 2015-05-26 11:55:17 -07:00
Kai Zheng
1a37b0fe9a HDFS-8074 Define a system-wide default EC schema. Contributed by Kai Zheng 2015-05-26 11:55:16 -07:00
Vinayakumar B
2ca3e38feb HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from NameNode (Contributed by Vinayakumar B)
Added missed file
2015-05-26 11:55:16 -07:00
Vinayakumar B
16484f0030 HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from NameNode (Contributed by Vinayakumar B) 2015-05-26 11:55:16 -07:00
Zhe Zhang
bff59392ee HDFS-7782. Erasure coding: pread from files in striped layout. Contributed by Zhe Zhang and Jing Zhao 2015-05-26 11:55:15 -07:00
Zhe Zhang
91c741a2a1 HDFS-7782. Erasure coding: pread from files in striped layout. Contributed by Zhe Zhang and Jing Zhao 2015-05-26 11:55:15 -07:00
Vinayakumar B
aac73c21c3 HADOOP-11740. Combine erasure encoder and decoder interfaces. Contributed by Zhe Zhang.
Updated CHANGES-HDFS-EC-7285.txt
2015-05-26 11:55:15 -07:00
Vinayakumar B
d25ca09cf6 HADOOP-11782 Correct two thrown messages in ECSchema class. Contributed by Xinwei Qin
Updated CHANGES-HDFS-EC-7285.txt
2015-05-26 11:55:14 -07:00
Zhe Zhang
146ce7a978 HDFS-7969. Erasure coding: NameNode support for lease recovery of striped block groups. Contributed by Zhe Zhang. 2015-05-26 11:55:14 -07:00
Zhe Zhang
c243319eab HDFS-7936. Erasure coding: resolving conflicts in the branch when merging trunk changes (this commit is for HDFS-8035). Contributed by Zhe Zhang 2015-05-26 11:55:13 -07:00
Zhe Zhang
e54a74b566 HADOOP-11740. Combine erasure encoder and decoder interfaces. Contributed by Zhe Zhang. 2015-05-26 11:55:13 -07:00
Zhe Zhang
1af8c14862 HDFS-7839. Erasure coding: implement facilities in NameNode to create and manage EC zones. Contributed by Zhe Zhang 2015-05-26 11:55:07 -07:00
Zhe Zhang
578019d6a2 HDFS-7936. Erasure coding: resolving conflicts in the branch when merging (this commit is for conflicts from HDFS-6945). Contributed by Zhe Zhang. 2015-05-26 11:44:47 -07:00
Zhe Zhang
9edc73d123 HDFS-7617. Add unit tests for editlog transactions for EC. Contributed by Hui Zheng. 2015-05-26 11:44:47 -07:00
Vinayakumar B
af8eaacdb4 HDFS-8027. Erasure Coding: Update CHANGES-HDFS-7285.txt with branch commits (Vinayakumar B) 2015-05-26 11:44:46 -07:00
Jing Zhao
cd655ee817 HDFS-8005. Erasure Coding: simplify striped block recovery work computation and add tests. Contributed by Jing Zhao. 2015-05-26 11:44:42 -07:00
Jing Zhao
abf833a7b2 HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped blocks in NameNode. Contributed by Jing Zhao. 2015-05-26 11:43:44 -07:00
Zhe Zhang
97378e4cd0 HDFS-7936. Erasure coding: resolving conflicts in the branch when merging (this commit is for HDFS-7742) 2015-05-26 11:43:43 -07:00
Tsz-Wo Nicholas Sze
9d1175b8fb HDFS-7716. Add a test for BlockGroup support in FSImage. Contributed by Takuya Fukudome 2015-05-26 11:43:43 -07:00
Zhe Zhang
ea2e60fbcc HDFS-7936. Erasure coding: resolving conflicts when merging with HDFS-7903, HDFS-7435, HDFS-7930, HDFS-7960 (this commit is for HDFS-7960) 2015-05-26 11:43:42 -07:00
Jing Zhao
d0d75a8339 HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage. Contributed by Hui Zheng. 2015-05-26 11:43:42 -07:00
Jing Zhao
46dac3595f HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks. Contributed by GAO Rui. 2015-05-26 11:43:42 -07:00
Zhe Zhang
544f75d651 HDFS-7936. Erasure coding: resolving conflicts when merging with HDFS-7903, HDFS-7435 and HDFS-7930 (this commit is for HDFS-7930 only) 2015-05-26 11:43:42 -07:00
Zhe Zhang
57a84c0d14 HDFS-7369. Erasure coding: distribute recovery work for striped blocks to DataNode. Contributed by Zhe Zhang. 2015-05-26 11:43:37 -07:00
Jing Zhao
a38a37c634 HDFS-7912. Erasure Coding: track BlockInfo instead of Block in UnderReplicatedBlocks and PendingReplicationBlocks. Contributed by Jing Zhao. 2015-05-26 11:41:59 -07:00
Jing Zhao
26773d9d6c HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped blocks. Contributed by Kai Sasaki. 2015-05-26 11:32:26 -07:00
Jing Zhao
11585883a9 HDFS-7936. Erasure coding: resolving conflicts when merging with HDFS-7903 and HDFS-7435. Contributed by Zhe Zhang. 2015-05-26 11:32:26 -07:00
Jing Zhao
f05c21285e HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from striped files. Contributed by Jing Zhao. 2015-05-26 11:32:25 -07:00
Jing Zhao
39a0a85fb7 HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print striped blocks. Contributed by Takuya Fukudome. 2015-05-26 11:32:25 -07:00
Jing Zhao
1e1e930407 HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode. Contributed by Jing Zhao. 2015-05-26 11:32:20 -07:00
Zhe Zhang
9af1f4779b Addendum fix for HDFS-7749 to be compatible with HDFS-7993 2015-05-26 11:07:16 -07:00
Jing Zhao
9f2f583f40 HDFS-7749. Erasure Coding: Add striped block support in INodeFile. Contributed by Jing Zhao. 2015-05-26 11:07:15 -07:00
Jing Zhao
ba93714920 HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info. Contributed by Jing Zhao. 2015-05-26 11:07:11 -07:00
Jing Zhao
ec48013426 Fix Compilation Error in TestAddBlockgroup.java after the merge 2015-05-26 11:03:34 -07:00
Zhe Zhang
0c1da5a030 HDFS-7652. Process block reports for erasure coded blocks. Contributed by Zhe Zhang 2015-05-26 11:03:34 -07:00
Zhe Zhang
bc2833b1c9 HDFS-7339. Allocating and persisting block groups in NameNode. Contributed by Zhe Zhang 2015-05-26 11:03:34 -07:00
Vinayakumar B
f166e67a23 HDFS-7347. Configurable erasure coding policy for individual files and directories ( Contributed by Zhe Zhang ) 2015-05-26 11:03:34 -07:00
Haohui Mai
ada233b7cd HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang. 2015-05-24 22:30:32 -07:00
Vinayakumar B
55ed6558ba HADOOP-11743. maven doesn't clean all the site files (Contributed by ramtin) 2015-05-22 16:45:19 +05:30
Vinayakumar B
0c6638c2ea HDFS-8268. Port conflict log for data node server is not sufficient (Contributed by Mohammad Shahid Khan) 2015-05-22 16:15:15 +05:30
Arpit Agarwal
cf2b5694d6 HDFS-8454. Remove unnecessary throttling in TestDatanodeDeath. (Arpit Agarwal) 2015-05-21 20:43:57 -07:00
Xiaoyu Yao
05e04f34f2 HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. 2015-05-21 11:58:00 -07:00
Haohui Mai
2b6bcfdafa HDFS-8421. Move startFile() and related functions into FSDirWriteFileOp. Contributed by Haohui Mai. 2015-05-21 08:08:28 -07:00
Akira Ajisaka
0e4f1081c7 HADOOP-10366. Add whitespaces between classes for values in core-default.xml to fit better in browser. Contributed by kanaka kumar avvaru. 2015-05-21 17:52:03 +09:00
Akira Ajisaka
dc8434ab2b HDFS-4383. Document the lease limits. Contributed by Arshad Mohammad. 2015-05-21 17:30:43 +09:00
Kihwal Lee
8860e352c3 HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. 2015-05-19 13:05:15 -05:00
Colin Patrick Mccabe
470c87dbc6 HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7 ThreadLocalRandom (Sean Busbey via Colin P. McCabe) 2015-05-19 10:50:15 -07:00
Kihwal Lee
de30d66b26 HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. 2015-05-19 08:04:38 -05:00
Vinayakumar B
d4a2830b63 HADOOP-11103. Clean up RemoteException (Contributed by Sean Busbey) 2015-05-19 14:41:05 +05:30
Vinayakumar B
93972a332a HDFS-6348. SecondaryNameNode not terminating properly on runtime exceptions (Contributed by Rakesh R) 2015-05-19 12:24:25 +05:30
Ravi Prakash
cdfae446ad HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) 2015-05-18 12:37:21 -07:00
Tsz-Wo Nicholas Sze
0c590e1c09 HDFS-8405. Fix a typo in NamenodeFsck. Contributed by Takanobu Asanuma 2015-05-19 02:57:54 +08:00
Arpit Agarwal
a2190bf15d HDFS-8345. Storage policy APIs must be exposed via the FileSystem interface. (Arpit Agarwal) 2015-05-18 11:36:29 -07:00
Uma Maheswara Rao G
a6af0248e9 HDFS-8412. Fix the test failures in HTTPFS: In some tests setReplication called after fs close. Contributed by Uma Maheswara Rao G. 2015-05-18 19:35:53 +05:30
Uma Maheswara Rao G
363c35541d Updating CHANGES.txt for moving entry of HDFS-8332 from branch-2 to trunk 2015-05-18 14:48:49 +05:30
Arpit Agarwal
e453989a57 HDFS-8157. Writes to RAM DISK reserve locked memory for block files. (Arpit Agarwal) 2015-05-16 09:05:35 -07:00
Haohui Mai
ac742c762d HDFS-8403. Eliminate retries in TestFileCreation#testOverwriteOpenForWrite. Contributed by Arpit Agarwal. 2015-05-15 19:12:40 -07:00
Haohui Mai
e5afac5896 HDFS-8394. Move getAdditionalBlock() and related functionalities into a separate class. Contributed by Haohui Mai. 2015-05-15 19:09:59 -07:00
Jing Zhao
8f37873342 HDFS-8397. Refactor the error handling code in DataStreamer. Contributed by Tsz Wo Nicholas Sze. 2015-05-15 16:14:54 -07:00
Vinayakumar B
3bef7c80a9 HDFS-6888. Allow selectively audit logging ops (Contributed by Chen He) 2015-05-15 11:07:51 +05:30
Akira Ajisaka
ee7beda6e3 HDFS-8350. Remove old webhdfs.xml and other outdated documentation stuff. Contributed by Brahma Reddy Battula. 2015-05-15 13:54:35 +09:00
Akira Ajisaka
cbc01ed08e HDFS-8371. Fix test failure in TestHdfsConfigFields for spanreceiver properties. Contributed by Ray Chiang. 2015-05-15 12:14:47 +09:00
Vinayakumar B
def9136e02 HDFS-8150. Make getFileChecksum fail for blocks under construction (Contributed by J.Andreina) 2015-05-14 15:54:51 +05:30
Haohui Mai
b2c85db86c HDFS-7728. Avoid updating quota usage while loading edits. Contributed by Jing Zhao. 2015-05-13 21:50:35 -07:00