big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao.

Browse Source
This commit is contained in:
Akira Ajisaka 2015-04-30 20:12:32 +09:00
parent f384a063a6
commit de9404f02f
3 changed files with 16 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -603,6 +603,9 @@ Release 2.7.1 - UNRELEASED
HDFS-8204. Mover/Balancer should not schedule two replicas to the same HDFS-8204. Mover/Balancer should not schedule two replicas to the same
datanode. (Walter Su via szetszwo) datanode. (Walter Su via szetszwo)
HDFS-7770. Need document for storage type label of data node storage
locations under dfs.data.dir. (Xiaoyu Yao via aajisaka)
OPTIMIZATIONS OPTIMIZATIONS
BUG FIXES BUG FIXES

7
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
View File

@ -430,8 +430,11 @@
<description>Determines where on the local filesystem an DFS data node <description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named list of directories, then data will be stored in all named
directories, typically on different devices. directories, typically on different devices. The directories should be tagged
Directories that do not exist are ignored. with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
storage policies. The default storage type will be DISK if the directory does
not have a storage type tagged explicitly. Directories that do not exist will
be created if local filesystem permission allows.
</description> </description>
</property> </property>

8
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
View File

@ -95,6 +95,14 @@ The effective storage policy can be retrieved by the "[`dfsadmin -getStoragePoli
### Configuration ### Configuration
* **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`. * **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`.
* **dfs.datanode.data.dir** - on each data node, the comma-separated storage locations should be tagged with their storage types. This allows storage policies to place the blocks on different storage types according to policy. For example:
1. A datanode storage location /grid/dn/disk0 on DISK should be configured with `[DISK]file:///grid/dn/disk0`
2. A datanode storage location /grid/dn/ssd0 on SSD can should configured with `[SSD]file:///grid/dn/ssd0`
3. A datanode storage location /grid/dn/archive0 on ARCHIVE should be configured with `[ARCHIVE]file:///grid/dn/archive0`
4. A datanode storage location /grid/dn/ram0 on RAM_DISK should be configured with `[RAM_DISK]file:///grid/dn/ram0`
The default storage type of a datanode storage location will be DISK if it does not have a storage type tagged explicitly.
Mover - A New Data Migration Tool Mover - A New Data Migration Tool
--------------------------------- ---------------------------------