big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12,835 Commits 2 Branches 2 Tags 264 MiB
dd9ebf6eed
Commit Graph

336 Commits

Author SHA1 Message Date
Colin Patrick Mccabe
dd9ebf6eed HDFS-9260. Improve the performance and GC friendliness of NameNode startup and full block reports (Staffan Friberg via cmccabe) 2016-02-02 11:23:00 -08:00
Jing Zhao
ec25c7f9c7 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 16:34:40 -08:00
Jing Zhao
3a9571308e Revert "HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu." 
This reverts commit 8a91109d16.
 2016-01-27 16:31:19 -08:00
Jing Zhao
8a91109d16 HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:48:47 -08:00
Tsz-Wo Nicholas Sze
70d6f20126 HDFS-9575. Use byte array for internal block indices in a striped block. Contributed by jing9 2015-12-22 14:47:57 +08:00
Zhe Zhang
61ab0440f7 HDFS-9173. Erasure Coding: Lease recovery for striped file. Contributed by Walter Su and Jing Zhao. 
Change-Id: I51703a61c9d8454f883028f3f6acb5729fde1b15
 2015-12-18 15:57:48 -08:00
Vinayakumar B
9f256d1d71 HDFS-9426. Rollingupgrade finalization is not backward compatible (Contributed by Kihwal Lee) 
(cherry picked from commit c62d42cd8bb09a5ffc0c5eefa2d87913e71b9e7e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
 2015-11-26 09:39:47 +05:30
Zhe Zhang
8fd5520246 Merge remote-tracking branch 'apache/trunk' into HDFS-7285 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFS.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java

Change-Id: Ic7946c4ea35bed587fe879ce58b959b25ecc0823
 2015-09-29 01:39:16 -07:00
Haohui Mai
7c5c099324 HDFS-8733. Keep server related definition in hdfs.proto on server side. Contributed by Mingliang Liu. 2015-09-22 20:57:05 -07:00
Zhe Zhang
6b6a63bbbd Merge commit '456e901a4c5c639267ee87b8e5f1319f256d20c2' (HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. Contributed by Haohui Mai.) into HDFS-7285-merge 2015-08-24 12:59:26 -07:00
Zhe Zhang
1d37a88121 HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs. Contributed by Walter Su. 2015-08-13 10:04:45 -07:00
Walter Su
06394e3760 HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock. Contributed by Walter Su. 2015-07-20 10:18:34 +08:00
Zhe Zhang
7e091de136 HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile. Contributed by Zhe Zhang and Yi Liu. 2015-07-15 09:49:32 -07:00
Haohui Mai
fc6182d5ed HDFS-8726. Move protobuf files that define the client-sever protocols to hdfs-client. Contributed by Haohui Mai. 2015-07-08 10:37:10 -07:00
Vinayakumar B
0b7af27b9a HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in double Audit log entries (Contributed by Vinayakumar B) 2015-06-30 11:02:13 +05:30
Colin Patrick Mccabe
d112d18324 HDFS-8582. Support getting a list of reconfigurable config properties and do not generate spurious reconfig warnings (Lei (Eddy) Xu via Colin P. McCabe) 2015-06-19 19:14:05 -07:00
Jing Zhao
c12a974ccf HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature. Contributed by Walter Su. 2015-06-17 11:35:31 -07:00
Jing Zhao
683332b36d HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from StripedBlockProto. Contributed by Yi Liu. 2015-06-12 14:48:53 -07:00
Colin Patrick Mccabe
12b5b06c06 HDFS-7923. The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages (cmccabe) 2015-06-12 11:17:51 -07:00
Zhe Zhang
673280df24 HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic. Contributed by Walter Su. 2015-06-03 11:51:58 -07:00
Vinayakumar B
9a18598e2d HDFS-8408. Revisit and refactor ErasureCodingInfo (Contributed by Vinayakumar B) 2015-05-26 12:07:15 -07:00
Zhe Zhang
91c81fdc24 HDFS-8375. Add cellSize as an XAttr to ECZone. Contributed by Vinayakumar B. 2015-05-26 12:02:33 -07:00
Jing Zhao
9da927540f HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. Contributed by Yong Zhang. 2015-05-26 12:01:49 -07:00
Uma Maheswara Rao G
ac97edd1ab HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding. Contributed by Uma Maheswara Rao G 2015-05-26 12:01:48 -07:00
Uma Maheswara Rao G
436c14855a HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command. Contributed by Uma Maheswara Rao G 2015-05-26 11:59:57 -07:00
Uma Maheswara Rao G
014d8675c5 HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration. (Contributed by Uma Maheswara Rao G) 2015-05-26 11:59:52 -07:00
Vinayakumar B
a1c9425265 HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for making it ready for transfer to DN (Contributed by Uma Maheswara Rao G) 2015-05-26 11:59:51 -07:00
Vinayakumar B
ceb3d1c170 HDFS-7349. Support DFS command for the EC encoding (Contributed by Vinayakumar B) 2015-05-26 11:59:11 -07:00
Vinayakumar B
04c23d95d0 HDFS-8123. Erasure Coding: Better to move EC related proto messages to a separate erasurecoding proto file (Contrubuted by Rakesh R) 2015-05-26 11:59:11 -07:00
Zhe Zhang
a32c4dc38a HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone. Contributed by Vinayakumar B. 2015-05-26 11:59:06 -07:00
Vinayakumar B
f53e402635 HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all ECSchemas loaded in Namenode. (Contributed by Vinayakumar B) 2015-05-26 11:58:40 -07:00
Vinayakumar B
16484f0030 HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from NameNode (Contributed by Vinayakumar B) 2015-05-26 11:55:16 -07:00
Zhe Zhang
1af8c14862 HDFS-7839. Erasure coding: implement facilities in NameNode to create and manage EC zones. Contributed by Zhe Zhang 2015-05-26 11:55:07 -07:00
Jing Zhao
f05c21285e HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from striped files. Contributed by Jing Zhao. 2015-05-26 11:32:25 -07:00
Jing Zhao
1e1e930407 HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode. Contributed by Jing Zhao. 2015-05-26 11:32:20 -07:00
Jing Zhao
9f2f583f40 HDFS-7749. Erasure Coding: Add striped block support in INodeFile. Contributed by Jing Zhao. 2015-05-26 11:07:15 -07:00
Jing Zhao
fddd55279d HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 2015-04-14 15:05:23 -07:00
Arpit Agarwal
1a495fbb48 HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times (Contributed by Vinayakumar B and Keisuke Ogiwara) 2015-03-30 15:25:16 -07:00
Haohui Mai
b80457158d HDFS-7748. Separate ECN flags from the Status in the DataTransferPipelineAck. Contributed by Anu Engineer and Haohui Mai. 2015-03-30 11:59:21 -07:00
Arpit Agarwal
3d0708bdb0 HDFS-7824. GetContentSummary API and its namenode implementation for Storage Type Quota/Usage. (Contributed by Xiaoyu Yao) 2015-03-26 10:24:11 -07:00
Jing Zhao
5e21e4ca37 HDFS-6353. Check and make checkpoint before stopping the NameNode. Contributed by Jing Zhao. 2015-03-25 10:38:00 -07:00
Andrew Wang
50ee8f4e67 HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 2015-03-23 22:00:34 -07:00
Kihwal Lee
75ead273be HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now(). Contributed by Vinayakumar B 2015-03-20 14:02:09 -05:00
Colin Patrick Mccabe
bc9cb3e271 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 2015-03-14 22:36:46 -07:00
Colin Patrick Mccabe
32741cf3d2 Revert "HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe)" (jenkins didn't run yet) 
This reverts commit 5aa892ed48.
 2015-03-13 18:40:20 -07:00
Colin Patrick Mccabe
5aa892ed48 HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 2015-03-13 18:29:49 -07:00
Kihwal Lee
d324164a51 HDFS-7435. PB encoding of block reports is very inefficient. Contributed by Daryn Sharp. 2015-03-13 14:23:37 -05:00
cnauroth
9729b244de HDFS-7604. Track and display failed DataNode storage locations in NameNode. Contributed by Chris Nauroth. 2015-02-16 14:43:02 -08:00
Arpit Agarwal
f2231cebcd HDFS-7775. Use consistent naming for NN-internal quota related types and functions. (Contributed bu Xiaoyu Yao) 2015-02-13 21:01:33 -08:00
Tsz-Wo Nicholas Sze
085b1e293f HDFS-6133. Add a feature for replica pinning so that a pinned replica will not be moved by Balancer/Mover. Contributed by zhaoyunjiong 2015-02-11 15:12:12 -08:00