big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw)

Browse Source
This commit is contained in:
Allen Wittenauer 2014-09-29 08:51:41 -07:00
parent 4666440b1a
commit 9829f9fa58
3 changed files with 71 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -516,6 +516,8 @@ Release 2.6.0 - UNRELEASED
HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal)
HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw)
OPTIMIZATIONS
HDFS-6690. Deduplicate xattr names in memory. (wang)

11
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
View File

@ -2140,4 +2140,15 @@
</description>
</property>
<property>
<name>dfs.namenode.legacy-oiv-image.dir</name>
<value></value>
<description>Determines where to save the namespace in the old fsimage format
during checkpointing by standby NameNode or SecondaryNameNode. Users can
dump the contents of the old format fsimage by oiv_legacy command. If
the value is not specified, old format fsimage will not be saved in
checkpoint.
</description>
</property>
</configuration>

59
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
View File

@ -28,7 +28,7 @@ Offline Image Viewer Guide
namespace. The tool is able to process very large image files relatively
quickly. The tool handles the layout formats that were included with Hadoop
versions 2.4 and up. If you want to handle older layout formats, you can
use the Offline Image Viewer of Hadoop 2.3.
use the Offline Image Viewer of Hadoop 2.3 or {{oiv_legacy Command}}.
If the tool is not able to process an image file, it will exit cleanly.
The Offline Image Viewer does not require a Hadoop cluster to be running;
it is entirely offline in its operation.
@ -188,3 +188,60 @@ Offline Image Viewer Guide
about the hdfs namespace. This information can then be used to explore
file system usage patterns or find specific files that match arbitrary
criteria, along with other types of namespace analysis.
* oiv_legacy Command
Due to the internal layout changes introduced by the ProtocolBuffer-based
fsimage ({{{https://issues.apache.org/jira/browse/HDFS-5698}HDFS-5698}}),
OfflineImageViewer consumes excessive amount of memory and loses some
functions such as Indented and Delimited processor. If you want to process
without large amount of memory or use these processors, you can use
<<<oiv_legacy>>> command (same as <<<oiv>>> in Hadoop 2.3).
** Usage
1. Set <<<dfs.namenode.legacy-oiv-image.dir>>> to an appropriate directory
to make standby NameNode or SecondaryNameNode save its namespace in the
old fsimage format during checkpointing.
2. Use <<<oiv_legacy>>> command to the old format fsimage.
----
bash$ bin/hdfs oiv_legacy -i fsimage_old -o output
----
** Options
*-----------------------:-----------------------------------+
| <<Flag>> | <<Description>> |
*-----------------------:-----------------------------------+
| <<<-i>>>\|<<<--inputFile>>> <input file> | Specify the input fsimage file to
| | process. Required.
*-----------------------:-----------------------------------+
| <<<-o>>>\|<<<--outputFile>>> <output file> | Specify the output filename, if
| | the specified output processor generates one. If the
| | specified file already exists, it is silently
| | overwritten. Required.
*-----------------------:-----------------------------------+
| <<<-p>>>\|<<<--processor>>> <processor> | Specify the image processor to
| | apply against the image file. Valid options are
| | Ls (default), XML, Delimited, Indented, and
| | FileDistribution.
*-----------------------:-----------------------------------+
| <<<-skipBlocks>>> | Do not enumerate individual blocks within files. This
| | may save processing time and outfile file space on
| | namespaces with very large files. The Ls processor
| | reads the blocks to correctly determine file sizes
| | and ignores this option.
*-----------------------:-----------------------------------+
| <<<-printToScreen>>> | Pipe output of processor to console as well as
| | specified file. On extremely large namespaces, this
| | may increase processing time by an order of
| | magnitude.
*-----------------------:-----------------------------------+
| <<<-delimiter>>> <arg>| When used in conjunction with the Delimited
| | processor, replaces the default tab delimiter with
| | the string specified by <arg>.
*-----------------------:-----------------------------------+
| <<<-h>>>\|<<<--help>>>| Display the tool usage and help information and exit.
*-----------------------:-----------------------------------+