73066dc958
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1440245 13f79535-47bb-0310-9956-ffa450edef68
419 lines
19 KiB
Plaintext
419 lines
19 KiB
Plaintext
~~ Licensed under the Apache License, Version 2.0 (the "License");
|
|
~~ you may not use this file except in compliance with the License.
|
|
~~ You may obtain a copy of the License at
|
|
~~
|
|
~~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~~
|
|
~~ Unless required by applicable law or agreed to in writing, software
|
|
~~ distributed under the License is distributed on an "AS IS" BASIS,
|
|
~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
~~ See the License for the specific language governing permissions and
|
|
~~ limitations under the License. See accompanying LICENSE file.
|
|
|
|
---
|
|
Offline Image Viewer Guide
|
|
---
|
|
---
|
|
${maven.build.timestamp}
|
|
|
|
Offline Image Viewer Guide
|
|
|
|
\[ {{{./index.html}Go Back}} \]
|
|
|
|
%{toc|section=1|fromDepth=0}
|
|
|
|
* Overview
|
|
|
|
The Offline Image Viewer is a tool to dump the contents of hdfs fsimage
|
|
files to human-readable formats in order to allow offline analysis and
|
|
examination of an Hadoop cluster's namespace. The tool is able to
|
|
process very large image files relatively quickly, converting them to
|
|
one of several output formats. The tool handles the layout formats that
|
|
were included with Hadoop versions 16 and up. If the tool is not able
|
|
to process an image file, it will exit cleanly. The Offline Image
|
|
Viewer does not require an Hadoop cluster to be running; it is entirely
|
|
offline in its operation.
|
|
|
|
The Offline Image Viewer provides several output processors:
|
|
|
|
[[1]] Ls is the default output processor. It closely mimics the format of
|
|
the lsr command. It includes the same fields, in the same order, as
|
|
lsr : directory or file flag, permissions, replication, owner,
|
|
group, file size, modification date, and full path. Unlike the lsr
|
|
command, the root path is included. One important difference
|
|
between the output of the lsr command this processor, is that this
|
|
output is not sorted by directory name and contents. Rather, the
|
|
files are listed in the order in which they are stored in the
|
|
fsimage file. Therefore, it is not possible to directly compare the
|
|
output of the lsr command this this tool. The Ls processor uses
|
|
information contained within the Inode blocks to calculate file
|
|
sizes and ignores the -skipBlocks option.
|
|
|
|
[[2]] Indented provides a more complete view of the fsimage's contents,
|
|
including all of the information included in the image, such as
|
|
image version, generation stamp and inode- and block-specific
|
|
listings. This processor uses indentation to organize the output
|
|
into a hierarchal manner. The lsr format is suitable for easy human
|
|
comprehension.
|
|
|
|
[[3]] Delimited provides one file per line consisting of the path,
|
|
replication, modification time, access time, block size, number of
|
|
blocks, file size, namespace quota, diskspace quota, permissions,
|
|
username and group name. If run against an fsimage that does not
|
|
contain any of these fields, the field's column will be included,
|
|
but no data recorded. The default record delimiter is a tab, but
|
|
this may be changed via the -delimiter command line argument. This
|
|
processor is designed to create output that is easily analyzed by
|
|
other tools, such as [36]Apache Pig. See the [37]Analyzing Results
|
|
section for further information on using this processor to analyze
|
|
the contents of fsimage files.
|
|
|
|
[[4]] XML creates an XML document of the fsimage and includes all of the
|
|
information within the fsimage, similar to the lsr processor. The
|
|
output of this processor is amenable to automated processing and
|
|
analysis with XML tools. Due to the verbosity of the XML syntax,
|
|
this processor will also generate the largest amount of output.
|
|
|
|
[[5]] FileDistribution is the tool for analyzing file sizes in the
|
|
namespace image. In order to run the tool one should define a range
|
|
of integers [0, maxSize] by specifying maxSize and a step. The
|
|
range of integers is divided into segments of size step: [0, s[1],
|
|
..., s[n-1], maxSize], and the processor calculates how many files
|
|
in the system fall into each segment [s[i-1], s[i]). Note that
|
|
files larger than maxSize always fall into the very last segment.
|
|
The output file is formatted as a tab separated two column table:
|
|
Size and NumFiles. Where Size represents the start of the segment,
|
|
and numFiles is the number of files form the image which size falls
|
|
in this segment.
|
|
|
|
* Usage
|
|
|
|
** Basic
|
|
|
|
The simplest usage of the Offline Image Viewer is to provide just an
|
|
input and output file, via the -i and -o command-line switches:
|
|
|
|
----
|
|
bash$ bin/hdfs oiv -i fsimage -o fsimage.txt
|
|
----
|
|
|
|
This will create a file named fsimage.txt in the current directory
|
|
using the Ls output processor. For very large image files, this process
|
|
may take several minutes.
|
|
|
|
One can specify which output processor via the command-line switch -p.
|
|
For instance:
|
|
|
|
----
|
|
bash$ bin/hdfs oiv -i fsimage -o fsimage.xml -p XML
|
|
----
|
|
|
|
or
|
|
|
|
----
|
|
bash$ bin/hdfs oiv -i fsimage -o fsimage.txt -p Indented
|
|
----
|
|
|
|
This will run the tool using either the XML or Indented output
|
|
processor, respectively.
|
|
|
|
One command-line option worth considering is -skipBlocks, which
|
|
prevents the tool from explicitly enumerating all of the blocks that
|
|
make up a file in the namespace. This is useful for file systems that
|
|
have very large files. Enabling this option can significantly decrease
|
|
the size of the resulting output, as individual blocks are not
|
|
included. Note, however, that the Ls processor needs to enumerate the
|
|
blocks and so overrides this option.
|
|
|
|
Example
|
|
|
|
Consider the following contrived namespace:
|
|
|
|
----
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:17 /anotherDir
|
|
-rw-r--r-- 3 theuser supergroup 286631664 2009-03-16 21:15 /anotherDir/biggerfile
|
|
-rw-r--r-- 3 theuser supergroup 8754 2009-03-16 21:17 /anotherDir/smallFile
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:11 /mapredsystem
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:11 /mapredsystem/theuser
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:11 /mapredsystem/theuser/mapredsystem
|
|
drwx-wx-wx - theuser supergroup 0 2009-03-16 21:11 /mapredsystem/theuser/mapredsystem/ip.redacted.com
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:12 /one
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:12 /one/two
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:16 /user
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 21:19 /user/theuser
|
|
----
|
|
|
|
Applying the Offline Image Processor against this file with default
|
|
options would result in the following output:
|
|
|
|
----
|
|
machine:hadoop-0.21.0-dev theuser$ bin/hdfs oiv -i fsimagedemo -o fsimage.txt
|
|
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:16 /
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:17 /anotherDir
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:11 /mapredsystem
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:12 /one
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:16 /user
|
|
-rw-r--r-- 3 theuser supergroup 286631664 2009-03-16 14:15 /anotherDir/biggerfile
|
|
-rw-r--r-- 3 theuser supergroup 8754 2009-03-16 14:17 /anotherDir/smallFile
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:11 /mapredsystem/theuser
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:11 /mapredsystem/theuser/mapredsystem
|
|
drwx-wx-wx - theuser supergroup 0 2009-03-16 14:11 /mapredsystem/theuser/mapredsystem/ip.redacted.com
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:12 /one/two
|
|
drwxr-xr-x - theuser supergroup 0 2009-03-16 14:19 /user/theuser
|
|
----
|
|
|
|
Similarly, applying the Indented processor would generate output that
|
|
begins with:
|
|
|
|
----
|
|
machine:hadoop-0.21.0-dev theuser$ bin/hdfs oiv -i fsimagedemo -p Indented -o fsimage.txt
|
|
|
|
FSImage
|
|
ImageVersion = -19
|
|
NamespaceID = 2109123098
|
|
GenerationStamp = 1003
|
|
INodes [NumInodes = 12]
|
|
Inode
|
|
INodePath =
|
|
Replication = 0
|
|
ModificationTime = 2009-03-16 14:16
|
|
AccessTime = 1969-12-31 16:00
|
|
BlockSize = 0
|
|
Blocks [NumBlocks = -1]
|
|
NSQuota = 2147483647
|
|
DSQuota = -1
|
|
Permissions
|
|
Username = theuser
|
|
GroupName = supergroup
|
|
PermString = rwxr-xr-x
|
|
...remaining output omitted...
|
|
----
|
|
|
|
* Options
|
|
|
|
*-----------------------:-----------------------------------+
|
|
| <<Flag>> | <<Description>> |
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-i>>>\|<<<--inputFile>>> <input file> | Specify the input fsimage file to
|
|
| | process. Required.
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-o>>>\|<<<--outputFile>>> <output file> | Specify the output filename, if the
|
|
| | specified output processor generates one. If the specified file already
|
|
| | exists, it is silently overwritten. Required.
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-p>>>\|<<<--processor>>> <processor> | Specify the image processor to apply
|
|
| | against the image file. Currently valid options are Ls (default), XML
|
|
| | and Indented..
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-skipBlocks>>> | Do not enumerate individual blocks within files. This may
|
|
| | save processing time and outfile file space on namespaces with very
|
|
| | large files. The Ls processor reads the blocks to correctly determine
|
|
| | file sizes and ignores this option.
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-printToScreen>>> | Pipe output of processor to console as well as specified
|
|
| | file. On extremely large namespaces, this may increase processing time
|
|
| | by an order of magnitude.
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-delimiter>>> <arg>| When used in conjunction with the Delimited processor,
|
|
| | replaces the default tab delimiter with the string specified by arg.
|
|
*-----------------------:-----------------------------------+
|
|
| <<<-h>>>\|<<<--help>>>| Display the tool usage and help information and exit.
|
|
*-----------------------:-----------------------------------+
|
|
|
|
* Analyzing Results
|
|
|
|
The Offline Image Viewer makes it easy to gather large amounts of data
|
|
about the hdfs namespace. This information can then be used to explore
|
|
file system usage patterns or find specific files that match arbitrary
|
|
criteria, along with other types of namespace analysis. The Delimited
|
|
image processor in particular creates output that is amenable to
|
|
further processing by tools such as [38]Apache Pig. Pig provides a
|
|
particularly good choice for analyzing these data as it is able to deal
|
|
with the output generated from a small fsimage but also scales up to
|
|
consume data from extremely large file systems.
|
|
|
|
The Delimited image processor generates lines of text separated, by
|
|
default, by tabs and includes all of the fields that are common between
|
|
constructed files and files that were still under constructed when the
|
|
fsimage was generated. Examples scripts are provided demonstrating how
|
|
to use this output to accomplish three tasks: determine the number of
|
|
files each user has created on the file system, find files were created
|
|
but have not accessed, and find probable duplicates of large files by
|
|
comparing the size of each file.
|
|
|
|
Each of the following scripts assumes you have generated an output file
|
|
using the Delimited processor named foo and will be storing the results
|
|
of the Pig analysis in a file named results.
|
|
|
|
** Total Number of Files for Each User
|
|
|
|
This script processes each path within the namespace, groups them by
|
|
the file owner and determines the total number of files each user owns.
|
|
|
|
----
|
|
numFilesOfEachUser.pig:
|
|
-- This script determines the total number of files each user has in
|
|
-- the namespace. Its output is of the form:
|
|
-- username, totalNumFiles
|
|
|
|
-- Load all of the fields from the file
|
|
A = LOAD '$inputFile' USING PigStorage('\t') AS (path:chararray,
|
|
replication:int,
|
|
modTime:chararray,
|
|
accessTime:chararray,
|
|
blockSize:long,
|
|
numBlocks:int,
|
|
fileSize:long,
|
|
NamespaceQuota:int,
|
|
DiskspaceQuota:int,
|
|
perms:chararray,
|
|
username:chararray,
|
|
groupname:chararray);
|
|
|
|
|
|
-- Grab just the path and username
|
|
B = FOREACH A GENERATE path, username;
|
|
|
|
-- Generate the sum of the number of paths for each user
|
|
C = FOREACH (GROUP B BY username) GENERATE group, COUNT(B.path);
|
|
|
|
-- Save results
|
|
STORE C INTO '$outputFile';
|
|
----
|
|
|
|
This script can be run against pig with the following command:
|
|
|
|
----
|
|
bin/pig -x local -param inputFile=../foo -param outputFile=../results ../numFilesOfEachUser.pig
|
|
----
|
|
|
|
The output file's content will be similar to that below:
|
|
|
|
----
|
|
bart 1
|
|
lisa 16
|
|
homer 28
|
|
marge 2456
|
|
----
|
|
|
|
** Files That Have Never Been Accessed
|
|
|
|
This script finds files that were created but whose access times were
|
|
never changed, meaning they were never opened or viewed.
|
|
|
|
----
|
|
neverAccessed.pig:
|
|
-- This script generates a list of files that were created but never
|
|
-- accessed, based on their AccessTime
|
|
|
|
-- Load all of the fields from the file
|
|
A = LOAD '$inputFile' USING PigStorage('\t') AS (path:chararray,
|
|
replication:int,
|
|
modTime:chararray,
|
|
accessTime:chararray,
|
|
blockSize:long,
|
|
numBlocks:int,
|
|
fileSize:long,
|
|
NamespaceQuota:int,
|
|
DiskspaceQuota:int,
|
|
perms:chararray,
|
|
username:chararray,
|
|
groupname:chararray);
|
|
|
|
-- Grab just the path and last time the file was accessed
|
|
B = FOREACH A GENERATE path, accessTime;
|
|
|
|
-- Drop all the paths that don't have the default assigned last-access time
|
|
C = FILTER B BY accessTime == '1969-12-31 16:00';
|
|
|
|
-- Drop the accessTimes, since they're all the same
|
|
D = FOREACH C GENERATE path;
|
|
|
|
-- Save results
|
|
STORE D INTO '$outputFile';
|
|
----
|
|
|
|
This script can be run against pig with the following command and its
|
|
output file's content will be a list of files that were created but
|
|
never viewed afterwards.
|
|
|
|
----
|
|
bin/pig -x local -param inputFile=../foo -param outputFile=../results ../neverAccessed.pig
|
|
----
|
|
|
|
** Probable Duplicated Files Based on File Size
|
|
|
|
This script groups files together based on their size, drops any that
|
|
are of less than 100mb and returns a list of the file size, number of
|
|
files found and a tuple of the file paths. This can be used to find
|
|
likely duplicates within the filesystem namespace.
|
|
|
|
----
|
|
probableDuplicates.pig:
|
|
-- This script finds probable duplicate files greater than 100 MB by
|
|
-- grouping together files based on their byte size. Files of this size
|
|
-- with exactly the same number of bytes can be considered probable
|
|
-- duplicates, but should be checked further, either by comparing the
|
|
-- contents directly or by another proxy, such as a hash of the contents.
|
|
-- The scripts output is of the type:
|
|
-- fileSize numProbableDuplicates {(probableDup1), (probableDup2)}
|
|
|
|
-- Load all of the fields from the file
|
|
A = LOAD '$inputFile' USING PigStorage('\t') AS (path:chararray,
|
|
replication:int,
|
|
modTime:chararray,
|
|
accessTime:chararray,
|
|
blockSize:long,
|
|
numBlocks:int,
|
|
fileSize:long,
|
|
NamespaceQuota:int,
|
|
DiskspaceQuota:int,
|
|
perms:chararray,
|
|
username:chararray,
|
|
groupname:chararray);
|
|
|
|
-- Grab the pathname and filesize
|
|
B = FOREACH A generate path, fileSize;
|
|
|
|
-- Drop files smaller than 100 MB
|
|
C = FILTER B by fileSize > 100L * 1024L * 1024L;
|
|
|
|
-- Gather all the files of the same byte size
|
|
D = GROUP C by fileSize;
|
|
|
|
-- Generate path, num of duplicates, list of duplicates
|
|
E = FOREACH D generate group AS fileSize, COUNT(C) as numDupes, C.path AS files;
|
|
|
|
-- Drop all the files where there are only one of them
|
|
F = FILTER E by numDupes > 1L;
|
|
|
|
-- Sort by the size of the files
|
|
G = ORDER F by fileSize;
|
|
|
|
-- Save results
|
|
STORE G INTO '$outputFile';
|
|
----
|
|
|
|
This script can be run against pig with the following command:
|
|
|
|
----
|
|
bin/pig -x local -param inputFile=../foo -param outputFile=../results ../probableDuplicates.pig
|
|
----
|
|
|
|
The output file's content will be similar to that below:
|
|
|
|
----
|
|
1077288632 2 {(/user/tennant/work1/part-00501),(/user/tennant/work1/part-00993)}
|
|
1077288664 4 {(/user/tennant/work0/part-00567),(/user/tennant/work0/part-03980),(/user/tennant/work1/part-00725),(/user/eccelston/output/part-03395)}
|
|
1077288668 3 {(/user/tennant/work0/part-03705),(/user/tennant/work0/part-04242),(/user/tennant/work1/part-03839)}
|
|
1077288698 2 {(/user/tennant/work0/part-00435),(/user/eccelston/output/part-01382)}
|
|
1077288702 2 {(/user/tennant/work0/part-03864),(/user/eccelston/output/part-03234)}
|
|
----
|
|
|
|
Each line includes the file size in bytes that was found to be
|
|
duplicated, the number of duplicates found, and a list of the
|
|
duplicated paths. Files less than 100MB are ignored, providing a
|
|
reasonable likelihood that files of these exact sizes may be
|
|
duplicates.
|