2014-11-06 04:33:26 +00:00
|
|
|
BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS
|
|
|
|
|
|
|
|
|
|
HDFS-7347. Configurable erasure coding policy for individual files and
|
2015-03-25 10:01:03 +00:00
|
|
|
directories ( Zhe Zhang via vinayakumarb )
|
|
|
|
|
|
2015-03-31 09:42:09 +00:00
|
|
|
HDFS-7339. Representing striped block groups in NameNode with hierarchical
|
|
|
|
|
naming protocol ( Zhe Zhang )
|
|
|
|
|
|
|
|
|
|
HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode
|
|
|
|
|
(Jing Zhao via Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print
|
|
|
|
|
striped blocks (Takuya Fukudome via jing9)
|
|
|
|
|
|
|
|
|
|
HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from
|
|
|
|
|
striped files (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped
|
|
|
|
|
blocks ( Kai Sasaki via jing9 )
|
|
|
|
|
|
|
|
|
|
HDFS-7912. Erasure Coding: track BlockInfo instead of Block in
|
|
|
|
|
UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7369. Erasure coding: distribute recovery work for striped blocks to
|
|
|
|
|
DataNode (Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks
|
|
|
|
|
(GAO Rui via jing9)
|
|
|
|
|
|
|
|
|
|
HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage
|
|
|
|
|
( Hui Zheng via jing9 )
|
|
|
|
|
|
|
|
|
|
HDFS-7616. Add a test for BlockGroup support in FSImage.
|
2015-03-25 10:01:03 +00:00
|
|
|
(Takuya Fukudome via szetszwo)
|
|
|
|
|
|
2015-03-31 09:42:09 +00:00
|
|
|
HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped
|
|
|
|
|
blocks in NameNode (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-8005. Erasure Coding: simplify striped block recovery work computation
|
2015-04-03 22:22:50 +00:00
|
|
|
and add tests (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7617. Add unit tests for editlog transactions for EC
|
|
|
|
|
(Hui Zheng via Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-7839. Erasure coding: implement facilities in NameNode to create and
|
2015-04-08 07:18:59 +00:00
|
|
|
manage EC zones (Zhe Zhang)
|
|
|
|
|
|
2015-04-15 06:53:07 +00:00
|
|
|
HDFS-7969. Erasure coding: NameNode support for lease recovery of striped
|
|
|
|
|
block groups. (Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-7782. Erasure coding: pread from files in striped layout.
|
|
|
|
|
(Zhe Zhang and Jing Zhao via Zhe Zhang)
|
|
|
|
|
|
2015-04-08 07:18:59 +00:00
|
|
|
HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from
|
2015-04-08 17:30:02 +00:00
|
|
|
NameNode (vinayakumarb)
|
2015-05-20 00:25:27 +00:00
|
|
|
|
2015-04-09 16:16:28 +00:00
|
|
|
HDFS-8074. Define a system-wide default EC schema. (Kai Zheng)
|
|
|
|
|
|
2015-04-15 06:53:07 +00:00
|
|
|
HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks.
|
|
|
|
|
(Jing Zhao and Zhe Zhang via Jing Zhao)
|
|
|
|
|
|
2015-04-10 17:03:37 +00:00
|
|
|
HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng)
|
|
|
|
|
|
2015-04-10 09:37:32 +00:00
|
|
|
HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng)
|
|
|
|
|
|
|
|
|
|
HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all
|
2015-04-15 06:39:16 +00:00
|
|
|
ECSchemas loaded in Namenode. (vinayakumarb)
|
|
|
|
|
|
2015-04-15 06:53:07 +00:00
|
|
|
HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone.
|
|
|
|
|
(Vinayakumar B via Zhe Zhang)
|
|
|
|
|
|
|
|
|
|
HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this
|
|
|
|
|
operation fails. (Rakesh R via Zhe Zhang)
|
|
|
|
|
|
2015-04-15 06:39:16 +00:00
|
|
|
HDFS-8123. Erasure Coding: Better to move EC related proto messages to a
|
2015-04-15 11:08:22 +00:00
|
|
|
separate erasurecoding proto file (Rakesh R via vinayakumarb)
|
|
|
|
|
|
2015-04-16 20:16:37 +00:00
|
|
|
HDFS-7349. Support DFS command for the EC encoding (vinayakumarb)
|
|
|
|
|
|
2015-04-20 09:34:49 +00:00
|
|
|
HDFS-8120. Erasure coding: created util class to analyze striped block groups.
|
|
|
|
|
(Contributed by Zhe Zhang and Li Bo via Jing Zhao)
|
|
|
|
|
|
2015-04-16 20:16:37 +00:00
|
|
|
HDFS-7994. Detect if resevered EC Block ID is already used during namenode
|
|
|
|
|
startup. (Hui Zheng via szetszwo)
|
2015-04-17 19:05:31 +00:00
|
|
|
|
|
|
|
|
HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz).
|
2015-04-18 17:50:45 +00:00
|
|
|
|
2015-04-20 09:34:49 +00:00
|
|
|
HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-7937. Erasure Coding: INodeFile quota computation unit tests.
|
|
|
|
|
(Kai Sasaki via Jing Zhao)
|
|
|
|
|
|
|
|
|
|
HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks.
|
|
|
|
|
(Jing Zhao)
|
|
|
|
|
|
2015-04-18 17:50:45 +00:00
|
|
|
HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for
|
|
|
|
|
making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb)
|
2015-04-20 09:34:49 +00:00
|
|
|
|
|
|
|
|
HDFS-8181. createErasureCodingZone sets retryCache state as false always
|
|
|
|
|
(Uma Maheswara Rao G via vinayakumarb)
|
2015-04-21 00:42:02 +00:00
|
|
|
|
|
|
|
|
HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error.
|
|
|
|
|
(szetszwo)
|
2015-04-22 03:56:39 +00:00
|
|
|
|
|
|
|
|
HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to
|
|
|
|
|
create BlockReader. (szetszwo via Zhe Zhang)
|
2015-05-20 00:25:27 +00:00
|
|
|
|
2015-04-22 04:03:07 +00:00
|
|
|
HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema
|
|
|
|
|
in FileSystemLinkResolver. (szetszwo via Zhe Zhang)
|
2015-04-22 14:00:14 +00:00
|
|
|
|
|
|
|
|
HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration.
|
|
|
|
|
(umamahesh)
|
2015-04-22 21:48:54 +00:00
|
|
|
|
|
|
|
|
HDFS-8156. Add/implement necessary APIs even we just have the system default
|
|
|
|
|
schema. (Kai Zheng via Zhe Zhang)
|
2015-04-23 16:19:12 +00:00
|
|
|
|
|
|
|
|
HDFS-8136. Client gets and uses EC schema when reads and writes a stripping
|
2015-04-23 22:43:04 +00:00
|
|
|
file. (Kai Sasaki via Kai Zheng)
|
|
|
|
|
|
|
|
|
|
HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last
|
|
|
|
|
stripe is at the block group boundary. (jing9)
|
2015-04-23 22:48:21 +00:00
|
|
|
|
|
|
|
|
HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream.
|
|
|
|
|
(Yi Liu via jing9)
|
2015-04-24 16:30:38 +00:00
|
|
|
|
|
|
|
|
HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause
|
|
|
|
|
block id conflicts (Jing Zhao via Zhe Zhang)
|
2015-04-25 05:36:15 +00:00
|
|
|
|
|
|
|
|
HDFS-8033. Erasure coding: stateful (non-positional) read from files in
|
|
|
|
|
striped layout (Zhe Zhang)
|
2015-04-28 08:44:33 +00:00
|
|
|
|
|
|
|
|
HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY
|
2015-04-28 08:54:17 +00:00
|
|
|
commands from standbynode if any (vinayakumarb)
|
|
|
|
|
|
|
|
|
|
HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated
|
2015-04-28 20:42:24 +00:00
|
|
|
as Idempotent (vinayakumarb)
|
|
|
|
|
|
|
|
|
|
HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open.
|
|
|
|
|
(Kai Sasaki via jing9)
|
2015-04-29 22:53:31 +00:00
|
|
|
|
|
|
|
|
HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream
|
|
|
|
|
(stateful read). (Jing Zhao via Zhe Zhang)
|
2015-04-30 06:49:52 +00:00
|
|
|
|
|
|
|
|
HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil.
|
|
|
|
|
(Zhe Zhang)
|
2015-04-30 07:13:32 +00:00
|
|
|
|
|
|
|
|
HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of
|
|
|
|
|
datastreamer threads. (Rakesh R via Zhe Zhang)
|
2015-05-01 02:42:29 +00:00
|
|
|
|
|
|
|
|
HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage()
|
|
|
|
|
when loading editlog. (jing9)
|
2015-05-01 22:59:58 +00:00
|
|
|
|
|
|
|
|
HDFS-7949. WebImageViewer need support file size calculation with striped
|
|
|
|
|
blocks. (Rakesh R via Zhe Zhang)
|
2015-05-04 18:24:35 +00:00
|
|
|
|
|
|
|
|
HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249.
|
|
|
|
|
(Zhe Zhang via jing9)
|
2015-05-04 21:44:58 +00:00
|
|
|
|
|
|
|
|
HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout.
|
|
|
|
|
(jing9)
|
2015-05-05 05:52:52 +00:00
|
|
|
|
|
|
|
|
HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh)
|
2015-05-05 06:24:30 +00:00
|
|
|
|
|
|
|
|
HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands
|
|
|
|
|
(Rakesh R via vinayakumarb)
|
2015-05-05 13:55:21 +00:00
|
|
|
|
|
|
|
|
HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via
|
|
|
|
|
umamahesh)
|
2015-05-20 00:25:27 +00:00
|
|
|
|
2015-05-05 23:26:49 +00:00
|
|
|
HDFS-7672. Handle write failure for stripping blocks and refactor the
|
|
|
|
|
existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo)
|
2015-05-05 23:33:56 +00:00
|
|
|
|
|
|
|
|
HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks.
|
|
|
|
|
(Yi Liu via Zhe Zhang)
|
2015-05-06 22:34:37 +00:00
|
|
|
|
|
|
|
|
HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng)
|
|
|
|
|
|
|
|
|
|
HDFS-8334. Erasure coding: rename DFSStripedInputStream related test
|
|
|
|
|
classes. (Zhe Zhang)
|
2015-05-07 10:56:01 +00:00
|
|
|
|
|
|
|
|
HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding
|
|
|
|
|
(umamahesh)
|
2015-05-07 18:06:40 +00:00
|
|
|
|
|
|
|
|
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
|
|
|
|
|
(Yi Liu via jing9)
|
2015-05-07 18:52:49 +00:00
|
|
|
|
|
|
|
|
HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via
|
|
|
|
|
jing9)
|
2015-05-08 20:56:56 +00:00
|
|
|
|
|
|
|
|
HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction.
|
|
|
|
|
(Tsz Wo Nicholas Sze via jing9)
|
2015-05-12 04:10:23 +00:00
|
|
|
|
|
|
|
|
HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread).
|
|
|
|
|
(Zhe Zhang)
|
2015-05-12 18:43:04 +00:00
|
|
|
|
|
|
|
|
HDFS-8372. Erasure coding: compute storage type quotas for striped files,
|
|
|
|
|
to be consistent with HDFS-8327. (Zhe Zhang via jing9)
|
2015-05-12 21:31:28 +00:00
|
|
|
|
|
|
|
|
HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be
|
|
|
|
|
handled properly (Rakesh R via zhz)
|
2015-05-13 00:48:56 +00:00
|
|
|
|
|
|
|
|
HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu)
|
2015-05-13 06:10:25 +00:00
|
|
|
|
|
|
|
|
HDFS-8195. Erasure coding: Fix file quota change when we complete/commit
|
|
|
|
|
the striped blocks. (Takuya Fukudome via zhz)
|
2015-05-13 07:13:39 +00:00
|
|
|
|
|
|
|
|
HDFS-8364. Erasure coding: fix some minor bugs in EC CLI
|
|
|
|
|
(Walter Su via vinayakumarb)
|
2015-05-14 05:57:48 +00:00
|
|
|
|
|
|
|
|
HDFS-8391. NN should consider current EC tasks handling count from DN while
|
|
|
|
|
assigning new tasks. (umamahesh)
|
2015-05-18 16:10:30 +00:00
|
|
|
|
|
|
|
|
HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng)
|
2015-05-18 22:08:30 +00:00
|
|
|
|
2015-05-19 06:59:23 +00:00
|
|
|
HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549)
|
|
|
|
|
|
2015-05-18 22:08:30 +00:00
|
|
|
HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe.
|
|
|
|
|
(Walter Su via jing9)
|
2015-05-19 02:06:34 +00:00
|
|
|
|
|
|
|
|
HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN.
|
|
|
|
|
(Yi Liu via jing9)
|
2015-05-19 05:55:27 +00:00
|
|
|
|
|
|
|
|
HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz)
|
2015-05-19 20:58:50 +00:00
|
|
|
|
|
|
|
|
HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue
|
|
|
|
|
configurable in DFSStripedOutputStream. (Li Bo)
|
2015-05-19 06:59:23 +00:00
|
|
|
|
|
|
|
|
HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker.
|
|
|
|
|
(Rakesh R via waltersu4549)
|
2015-05-19 20:58:50 +00:00
|
|
|
|
|
|
|
|
HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz).
|
2015-05-20 00:25:27 +00:00
|
|
|
|
|
|
|
|
HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file.
|
|
|
|
|
(Yi Liu via zhz).
|
2015-05-20 04:19:51 +00:00
|
|
|
|
|
|
|
|
HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream.
|
|
|
|
|
(Tsz Wo Nicholas Sze via jing9)
|
2015-05-20 21:52:51 +00:00
|
|
|
|
|
|
|
|
HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped.
|
|
|
|
|
(Kai Sasaki via jing9)
|
2015-05-20 22:37:50 +00:00
|
|
|
|
|
|
|
|
HDFS-8186. Erasure coding: Make block placement policy for EC file configurable.
|
|
|
|
|
(Walter Su via zhz)
|
2015-05-21 21:40:14 +00:00
|
|
|
|
|
|
|
|
HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding.
|
|
|
|
|
(Rakesh R via zhz)
|
2015-05-22 02:25:53 +00:00
|
|
|
|
|
|
|
|
HDFS-8441. Erasure Coding: make condition check earlier for setReplication.
|
|
|
|
|
(waltersu4549)
|
2015-05-24 15:07:34 +00:00
|
|
|
|
|
|
|
|
HDFS-7768. Change fsck to support EC files. (Takanobu Asanuma via szetszwo)
|
2015-05-25 08:13:29 +00:00
|
|
|
|
|
|
|
|
HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng)
|
2015-05-25 06:27:17 +00:00
|
|
|
|
|
|
|
|
HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb)
|
2015-05-26 23:06:50 +00:00
|
|
|
|
|
|
|
|
HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to
|
|
|
|
|
sync with HDFS-8421. (Zhe Zhang via jing9)
|
2015-06-01 16:37:29 +00:00
|
|
|
|
|
|
|
|
HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks
|
|
|
|
|
recovering. (zhz)
|
2015-06-02 11:08:43 +00:00
|
|
|
|
|
|
|
|
HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin
|
|
|
|
|
(Uma Maheswara Rao G via vinayakumarb)
|
