diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 0997058f97..73399348f7 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -13,6 +13,10 @@ Trunk (Unreleased) HDFS-3125. Add JournalService to enable Journal Daemon. (suresh) + HDFS-2832. Heterogeneous Storages support in HDFS phase 1 - treat DataNode + as a collection of storages (see breakdown of tasks below for features and + contributors). + IMPROVEMENTS HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common. @@ -440,6 +444,131 @@ Trunk (Unreleased) HDFS-5626. dfsadmin -report shows incorrect cache values. (cmccabe) + BREAKDOWN OF HDFS-2832 SUBTASKS AND RELATED JIRAS + + HDFS-4985. Add storage type to the protocol and expose it in block report + and block locations. (Arpit Agarwal) + + HDFS-5115. Make StorageID a UUID. (Arpit Agarwal) + + HDFS-5000. DataNode configuration should allow specifying storage type. + (Arpit Agarwal) + + HDFS-4987. Namenode changes to track multiple storages per datanode. + (szetszwo) + + HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987. + (Junping Du via szetszwo) + + HDFS-5009. Include storage information in the LocatedBlock. (szetszwo) + + HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and + firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and + fix a synchronization problem in DatanodeStorageInfo. (szetszwo) + + HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo) + + HDFS-4990. Change BlockPlacementPolicy to choose storages instead of + datanodes. (szetszwo) + + HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal) + + HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal) + + HDFS-5222. Move block schedule information from DatanodeDescriptor to + DatanodeStorageInfo. (szetszwo) + + HDFS-4988. Datanode must support all the volumes as individual storages. + (Arpit Agarwal) + + HDFS-5377. Heartbeats from Datandode should include one storage report + per storage directory. (Arpit Agarwal) + + HDFS-5398. NameNode changes to process storage reports per storage + directory. (Arpit Agarwal) + + HDFS-5390. Send one incremental block report per storage directory. + (Arpit Agarwal) + + HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal) + + HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo) + + HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal) + + HDFS-5435. File append fails to initialize storageIDs. (Junping Du via + Arpit Agarwal) + + HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit + Agarwal) + + HDFS-5447. Fix TestJspHelper. (Arpit Agarwal) + + HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter. + + HDFS-5448. Datanode should generate its ID on first registration. (Arpit + Agarwal) + + HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit + Agarwal) + + HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal) + + HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry. + (Contributed by szetszwo) + + HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed + by szetszwo) + + HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit + Agarwal) + + HDFS-5470. Add back trunk's reportDiff algorithm to the branch. + (Contributed by szetszwo) + + HDFS-5472. Fix TestDatanodeManager, TestSafeMode and + TestNNThroughputBenchmark (Contributed by szetszwo) + + HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit + Agarwal) + + HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed + by Junping Du) + + HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo) + + HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal) + + HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal) + + HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal) + + HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo) + + HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit + Agarwal) + + HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch. + (Junping Du via Arpit Agarwal) + + HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal) + + HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit + Agarwal) + + HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit + Agarwal) + + HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by + szetszwo) + + HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo) + + HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is + not accurate. (Eric Sirianni via Arpit Agarwal) + + HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal) + Release 2.4.0 - UNRELEASED INCOMPATIBLE CHANGES diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES_HDFS-2832.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES_HDFS-2832.txt deleted file mode 100644 index 1391c14f82..0000000000 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES_HDFS-2832.txt +++ /dev/null @@ -1,127 +0,0 @@ - BREAKDOWN OF HDFS-2832 ENABLE SUPPORT FOR HETEROGENEOUS STORAGES IN HDFS - -IMPROVEMENTS: - - HDFS-4985. Add storage type to the protocol and expose it in block report - and block locations. (Arpit Agarwal) - - HDFS-5115. Make StorageID a UUID. (Arpit Agarwal) - - HDFS-5000. DataNode configuration should allow specifying storage type. - (Arpit Agarwal) - - HDFS-4987. Namenode changes to track multiple storages per datanode. - (szetszwo) - - HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987. - (Junping Du via szetszwo) - - HDFS-5009. Include storage information in the LocatedBlock. (szetszwo) - - HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and - firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and - fix a synchronization problem in DatanodeStorageInfo. (szetszwo) - - HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo) - - HDFS-4990. Change BlockPlacementPolicy to choose storages instead of - datanodes. (szetszwo) - - HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal) - - HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal) - - HDFS-5222. Move block schedule information from DatanodeDescriptor to - DatanodeStorageInfo. (szetszwo) - - HDFS-4988. Datanode must support all the volumes as individual storages. - (Arpit Agarwal) - - HDFS-5377. Heartbeats from Datandode should include one storage report - per storage directory. (Arpit Agarwal) - - HDFS-5398. NameNode changes to process storage reports per storage - directory. (Arpit Agarwal) - - HDFS-5390. Send one incremental block report per storage directory. - (Arpit Agarwal) - - HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal) - - HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo) - - HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal) - - HDFS-5435. File append fails to initialize storageIDs. (Junping Du via - Arpit Agarwal) - - HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit - Agarwal) - - HDFS-5447. Fix TestJspHelper. (Arpit Agarwal) - - HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter. - - HDFS-5448. Datanode should generate its ID on first registration. (Arpit - Agarwal) - - HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit - Agarwal) - - HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal) - - HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry. - (Contributed by szetszwo) - - HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed - by szetszwo) - - HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit - Agarwal) - - HDFS-5470. Add back trunk's reportDiff algorithm to the branch. - (Contributed by szetszwo) - - HDFS-5472. Fix TestDatanodeManager, TestSafeMode and - TestNNThroughputBenchmark (Contributed by szetszwo) - - HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit - Agarwal) - - HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed - by Junping Du) - - HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo) - - HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal) - - HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal) - - HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal) - - HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo) - - HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit - Agarwal) - - HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch. - (Junping Du via Arpit Agarwal) - - HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal) - - HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit - Agarwal) - - HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit - Agarwal) - - HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by - szetszwo) - - HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo) - - HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is - not accurate. (Eric Sirianni via Arpit Agarwal) - - HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal) -