- This document describes how a superuser can submit jobs or access hdfs on behalf of another user in a secured way. -
-- The code example described in the next section is applicable for the following use case. -
-- A superuser with username 'super' wants to submit job and access hdfs on behalf of a user joe. The superuser has kerberos credentials but user joe doesn't have any. The tasks are required to run as user joe and any file accesses on namenode are required to be done as user joe. It is required that user joe can connect to the namenode or job tracker on a connection authenticated with super's kerberos credentials. In other words super is impersonating the user joe. -
-- In this example super's kerberos credentials are used for login and a proxy user ugi object is created for joe. The operations are performed within the doAs method of this proxy user ugi object. -
-- The superuser must be configured on namenode and jobtracker to be allowed to impersonate another user. Following configurations are required. -
-- If these configurations are not present, impersonation will not be allowed and connection will fail. -
-
- If more lax security is preferred, the wildcard value * may be used to allow impersonation from any host or of any user.
-
- The superuser must have kerberos credentials to be able to impersonate another user. It cannot use delegation tokens for this feature. It would be wrong if superuser adds its own delegation token to the proxy user ugi, as it will allow the proxy user to connect to the service with the privileges of the superuser. -
-- However, if the superuser does want to give a delegation token to joe, it must first impersonate joe and get a delegation token for joe, in the same way as the code example above, and add it to the ugi of joe. In this way the delegation token will have the owner as joe. -
-- This document describes the standard deployment layout for Hadoop. With increased complexity and evolving Hadoop ecosystem, having standard deployment layout ensures better integration between Hadoop sub-projects. By making the installation process easier, we can lower the barrier to entry and increase Hadoop adoption. -
-- We need to divide Hadoop up into packages that can be independently upgraded. The list of packages should include: -
-Packages from other teams will include:
-These packages should be deployable with RPM on RedHat. We also need a package that depends on a version of each of these packages. In general, we can generate tarballs in the new deployment layout.
-Note that some packages, like Pig, which are user facing, will have 2 versions installed in a given deployment. This will be accomplished by modifying the package name and the associated binaries to include the version number.
-All of the following paths are based on a prefix directory that is the root of the installation. Our packages must support having multiple Hadoop stack installation on a computer at the same time. For RPMs, this means that the packages must be relocatable and honor the --prefix option.
-It is important to have a standard deployment that results from installing the packages regardless of the package manager. Here are the top level directories and a sample of what would be under each. Note that all of the packages are installed "flattened" into the prefix directory. For compatibility reasons, we should create "share/hadoop" that matches the old HADOOP_PREFIX and set the HADOOP_PREFIX variable to that.
-Note that we must continue to honor HADOOP_CONF_DIR to override the configuration location, but that it should default to $prefix/etc. User facing binaries and scripts go into bin. Configuration files go into etc with multiple configuration files having a directory. JNI shared libraries go into lib/jni/$tool since Java does not allow to specify the version of the library to load. Libraries that aren't loaded via System.loadLibrary are placed directly under lib. 64 bit versions of the libraries for platforms that support them should be placed in lib64. All of the architecture-independent pieces, including the jars for each tool will be placed in share/$tool. The default location for all the run time information will be in var. The storage will be in var/lib, the logs in var/log and the pid files in var/run.
-Path can be configured at compile phase or installation phase. For RPM, it takes advantage of the --relocate directive to allow path reconfiguration at install phase. For Debian package, path is configured at compile phase. -
-Build phase parameter:
-Install phase parameter:
-This guide describes the native hadoop library and includes a small discussion about native shared libraries.
- -Note: Depending on your environment, the term "native libraries" could - refer to all *.so's you need to compile; and, the term "native compression" could refer to all *.so's - you need to compile that are specifically related to compression. - Currently, however, this document only addresses the native hadoop library (libhadoop.so).
- -Hadoop has native implementations of certain components for - performance reasons and for non-availability of Java implementations. These - components are available in a single, dynamically-linked native library called - the native hadoop library. On the *nix platforms the library is named libhadoop.so.
- -It is fairly easy to use the native hadoop library:
- -The native hadoop library includes two components, the zlib and gzip - - compression codecs: -
- -The native hadoop library is imperative for gzip to work.
-The native hadoop library is supported on *nix platforms only. - The library does not to work with Cygwin - or the Mac OS X platform.
- -The native hadoop library is mainly used on the GNU/Linus platform and - has been tested on these distributions:
- - -On all the above distributions a 32/64 bit native hadoop library will work - with a respective 32/64 bit jvm.
-The pre-built 32-bit i386-Linux native hadoop library is available as part of the
- hadoop distribution and is located in the lib/native directory. You can download the
- hadoop distribution from Hadoop Common Releases.
Be sure to install the zlib and/or gzip development packages - whichever compression - codecs you want to use with your deployment.
-The native hadoop library is written in ANSI C - and is built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). - This means it should be straight-forward to build the library on any platform with a standards-compliant - C compiler and the GNU autotools-chain (see the supported platforms).
- -The packages you need to install on the target platform are:
-Once you installed the prerequisite packages use the standard hadoop build.xml
- file and pass along the compile.native flag (set to true) to build the native hadoop library:
$ ant -Dcompile.native=true <target>
You should see the newly-built library in:
- -$ build/native/<platform>/lib
where <platform> is a combination of the system-properties:
- ${os.name}-${os.arch}-${sun.arch.data.model} (for example, Linux-i386-32).
Please note the following:
-The bin/hadoop script ensures that the native hadoop
- library is on the library path via the system property:
- -Djava.library.path=<path>
During runtime, check the hadoop log files for your MapReduce tasks.
- - DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library... INFO util.NativeCodeLoader - Loaded the native-hadoop library
- INFO util.NativeCodeLoader - Unable to load native-hadoop library for
- your platform... using builtin-java classes where applicable
-
-
- You can load any native shared library using - DistributedCache - for distributing and symlinking the library files.
- -This example shows you how to distribute a shared library, mylib.so,
- and load it from a MapReduce task.
bin/hadoop fs -copyFromLocal mylib.so.1 /libraries/mylib.so.1
- DistributedCache.createSymlink(conf); DistributedCache.addCacheFile("hdfs://host:port/libraries/mylib.so.1#mylib.so", conf);
-
- System.loadLibrary("mylib.so");
-
Note: If you downloaded or built the native hadoop library, you don’t need to use DistibutedCache to
- make the library available to your MapReduce tasks.
This document describes how to configure and manage Service Level - Authorization for Hadoop.
-Make sure Hadoop is installed, configured and setup correctly. For more information see:
-Service Level Authorization is the initial authorization mechanism to - ensure clients connecting to a particular Hadoop service have the - necessary, pre-configured, permissions and are authorized to access the given - service. For example, a MapReduce cluster can use this mechanism to allow a - configured list of users/groups to submit jobs.
- -The ${HADOOP_CONF_DIR}/hadoop-policy.xml configuration file
- is used to define the access control lists for various Hadoop services.
Service Level Authorization is performed much before to other access - control checks such as file-permission checks, access control on job queues - etc.
-This section describes how to configure service-level authorization
- via the configuration file {HADOOP_CONF_DIR}/hadoop-policy.xml.
-
By default, service-level authorization is disabled for Hadoop. To
- enable it set the configuration property
- hadoop.security.authorization to true
- in ${HADOOP_CONF_DIR}/core-site.xml.
This section lists the various Hadoop services and their configuration - knobs:
- -| Property | -Service | -
|---|---|
security.client.protocol.acl |
- ACL for ClientProtocol, which is used by user code via the - DistributedFileSystem. | -
security.client.datanode.protocol.acl |
- ACL for ClientDatanodeProtocol, the client-to-datanode protocol - for block recovery. | -
security.datanode.protocol.acl |
- ACL for DatanodeProtocol, which is used by datanodes to - communicate with the namenode. | -
security.inter.datanode.protocol.acl |
- ACL for InterDatanodeProtocol, the inter-datanode protocol - for updating generation timestamp. | -
security.namenode.protocol.acl |
- ACL for NamenodeProtocol, the protocol used by the secondary - namenode to communicate with the namenode. | -
security.refresh.policy.protocol.acl |
- ACL for RefreshAuthorizationPolicyProtocol, used by the - dfsadmin and mradmin commands to refresh the security policy in-effect. - | -
security.ha.service.protocol.acl |
- ACL for HAService protocol used by HAAdmin to manage the - active and stand-by states of namenode. - | -
${HADOOP_CONF_DIR}/hadoop-policy.xml defines an access
- control list for each Hadoop service. Every access control list has a
- simple format:
The list of users and groups are both comma separated list of names. - The two lists are separated by a space.
- -Example: user1,user2 group1,group2.
Add a blank at the beginning of the line if only a list of groups - is to be provided, equivalently a comman-separated list of users followed - by a space or nothing implies only a set of given users.
- -A special value of * implies that all users are - allowed to access the service.
-The service-level authorization configuration for the NameNode and
- JobTracker can be changed without restarting either of the Hadoop master
- daemons. The cluster administrator can change
- ${HADOOP_CONF_DIR}/hadoop-policy.xml on the master nodes and
- instruct the NameNode and JobTracker to reload their respective
- configurations via the -refreshServiceAcl switch to
- dfsadmin and mradmin commands respectively.
Refresh the service-level authorization configuration for the - NameNode:
-
- $ bin/hadoop dfsadmin -refreshServiceAcl
-
Refresh the service-level authorization configuration for the - JobTracker:
-
- $ bin/hadoop mradmin -refreshServiceAcl
-
Of course, one can use the
- security.refresh.policy.protocol.acl property in
- ${HADOOP_CONF_DIR}/hadoop-policy.xml to restrict access to
- the ability to refresh the service-level authorization configuration to
- certain users/groups.
Allow only users alice, bob and users in the
- mapreduce group to submit jobs to the MapReduce cluster:
Allow only DataNodes running as the users who belong to the
- group datanodes to communicate with the NameNode:
Allow any user to talk to the HDFS cluster as a DFSClient:
- -This document describes how to set up and configure a single-node Hadoop - installation so that you can quickly perform simple operations using Hadoop - MapReduce and the Hadoop Distributed File System (HDFS).
- -Required software for Linux and Windows include:
-Additional requirements for Windows include:
-If your cluster doesn't have the requisite software you will need to - install it.
- -For example on Ubuntu Linux:
-
- $ sudo apt-get install ssh
- $ sudo apt-get install rsync
-
On Windows, if you did not install the required software when you - installed cygwin, start the cygwin installer and select the packages:
-- To get a Hadoop distribution, download a recent - stable release from one of the Apache Download - Mirrors. -
-
- Unpack the downloaded Hadoop distribution. In the distribution, edit the
- file conf/hadoop-env.sh to define at least
- JAVA_HOME to be the root of your Java installation.
-
- Try the following command:
- $ bin/hadoop
- This will display the usage documentation for the hadoop
- script.
-
Now you are ready to start your Hadoop cluster in one of the three supported - modes: -
-By default, Hadoop is configured to run in a non-distributed - mode, as a single Java process. This is useful for debugging.
- -
- The following example copies the unpacked conf directory to
- use as input and then finds and displays every match of the given regular
- expression. Output is written to the given output directory.
-
- $ mkdir input
- $ cp conf/*.xml input
-
- $ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
-
- $ cat output/*
-
Hadoop can also be run on a single-node in a pseudo-distributed mode - where each Hadoop daemon runs in a separate Java process.
- -Use the following:
-
- conf/core-site.xml:
conf/hdfs-site.xml:
conf/mapred-site.xml:
- Now check that you can ssh to the localhost without a passphrase:
- $ ssh localhost
-
- If you cannot ssh to localhost without a passphrase, execute the
- following commands:
- $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
- $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
-
- Format a new distributed-filesystem:
- $ bin/hadoop namenode -format
-
- Start the hadoop daemons:
- $ bin/start-all.sh
-
The hadoop daemon log output is written to the
- ${HADOOP_LOG_DIR} directory (defaults to
- ${HADOOP_PREFIX}/logs).
Browse the web interface for the NameNode and the JobTracker; by - default they are available at:
-NameNode -
- http://localhost:50070/
- JobTracker -
- http://localhost:50030/
-
- Copy the input files into the distributed filesystem:
- $ bin/hadoop fs -put conf input
-
- Run some of the examples provided:
-
- $ bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'
-
-
Examine the output files:
-
- Copy the output files from the distributed filesystem to the local
- filesytem and examine them:
- $ bin/hadoop fs -get output output
- $ cat output/*
-
or
-
- View the output files on the distributed filesystem:
- $ bin/hadoop fs -cat output/*
-
- When you're done, stop the daemons with:
- $ bin/stop-all.sh
-
For information on setting up fully-distributed, non-trivial clusters - see Cluster Setup.
-- Java and JNI are trademarks or registered trademarks of - Sun Microsystems, Inc. in the United States and other countries. -
- - - -- The synthetic load generator (SLG) is a tool for testing NameNode behavior - under different client loads. The user can generate different mixes - of read, write, and list requests by specifying the probabilities of - read and write. The user controls the intensity of the load by adjusting - parameters for the number of worker threads and the delay between - operations. While load generators are running, the user can profile and - monitor the running of the NameNode. When a load generator exits, it - prints some NameNode statistics like the average execution time of each - kind of operation and the NameNode throughput. -
-- The synopsis of the command is: -
-Options include:
- --readProbability <read probability>-writeProbability <write probability>-root <test space root>-maxDelayBetweenOps <maxDelayBetweenOpsInMillis>-numOfThreads <numOfThreads>-elapsedTime <elapsedTimeInSecs>-startTime <startTimeInMillis>-seed <seed>- After command line argument parsing, the load generator traverses - the test space and builds a table of all directories and another table - of all files in the test space. It then waits until the start time to - spawn the number of worker threads as specified by the user. - - Each thread sends a stream of requests to NameNode. At each iteration, - it first decides if it is going to read a file, create a file, or - list a directory following the read and write probabilities specified - by the user. The listing probability is equal to - 1-read probability-write probability. When reading, - it randomly picks a file in the test space and reads the entire file. - When writing, it randomly picks a directory in the test space and - creates a file there. -
-- To avoid two threads with the same load - generator or from two different load generators creating the same - file, the file name consists of the current machine's host name - and the thread id. The length of the file follows Gaussian - distribution with an average size of 2 blocks and the standard - deviation of 1. The new file is filled with byte 'a'. To avoid the test - space growing indefinitely, the file is deleted immediately - after the file creation completes. While listing, it randomly picks - a directory in the test space and lists its content. -
-- After an operation completes, the thread pauses for a random - amount of time in the range of [0, maxDelayBetweenOps] if the - specified maximum delay is not zero. All threads are stopped when - the specified elapsed time is passed. Before exiting, the program - prints the average execution for each kind of NameNode operations, - and the number of requests served by the NameNode per second. -
- -- The user needs to populate a test space before running a - load generator. The structure generator generates a random - test space structure and the data generator creates the files - and directories of the test space in Hadoop distributed file system. -
- -- This tool generates a random namespace structure with the - following constraints: -
- -- The generated namespace structure is described by two files in - the output directory. Each line of the first file contains the - full name of a leaf directory. Each line of the second file - contains the full name of a file and its size, separated by a blank. -
-- The synopsis of the command is: -
-Options include:
--maxDepth <maxDepth>-minWidth <minWidth>-maxWidth <maxWidth>-numOfFiles <#OfFiles>-avgFileSize <avgFileSizeInBlocks>-outDir <outDir>-seed <seed>- This tool reads the directory structure and file structure from - the input directory and creates the namespace in Hadoop distributed - file system. All files are filled with byte 'a'. -
-- The synopsis of the command is: -
-Options include:
--inDir <inDir>-root <test space root>This guide provides an overview of the Hadoop Fault Injection (FI) framework for those - who will be developing their own faults (aspects). -
-The idea of fault injection is fairly simple: it is an - infusion of errors and exceptions into an application's logic to - achieve a higher coverage and fault tolerance of the system. - Different implementations of this idea are available today. - Hadoop's FI framework is built on top of Aspect Oriented Paradigm - (AOP) implemented by AspectJ toolkit. -
-The current implementation of the FI framework assumes that the faults it - will be emulating are of non-deterministic nature. That is, the moment - of a fault's happening isn't known in advance and is a coin-flip based. -
-This piece of the FI framework allows you to set expectations for faults to happen. - The settings can be applied either statically (in advance) or in runtime. - The desired level of faults in the framework can be configured two ways: -
-src/aop/fi-site.xml
- configuration file. This file is similar to other Hadoop's config
- files
- build.properties
- file
- This is fundamentally a coin flipper. The methods of this class are - getting a random number between 0.0 - and 1.0 and then checking if a new number has happened in the - range of 0.0 and a configured level for the fault in question. If that - condition is true then the fault will occur. -
-Thus, to guarantee the happening of a fault one needs to set an - appropriate level to 1.0. - To completely prevent a fault from happening its probability level - has to be set to 0.0. -
-Note: The default probability level is set to 0
- (zero) unless the level is changed explicitly through the
- configuration file or in the runtime. The name of the default
- level's configuration parameter is
- fi.*
-
The foundation of Hadoop's FI framework includes a - cross-cutting concept implemented by AspectJ. The following basic - terms are important to remember: -
-- The following readily available join points are provided by AspectJ: -
-The aspect has two main parts:
-pointcut callReceivepacket()
- which servers as an identification mark of a specific point (in control
- and/or data flow) in the life of an application. before () throws IOException : callReceivepacket()
- - will be injected (see
- Putting It All Together)
- before that specific spot of the application's code.The pointcut identifies an invocation of class'
- java.io.OutputStream write()
- method
- with any number of parameters and any return type. This invoke should
- take place within the body of method
- receivepacket()
- from classBlockReceiver.
- The method can have any parameters and any return type.
- Possible invocations of
- write()
- method happening anywhere within the aspect
- BlockReceiverAspects
- or its heirs will be ignored.
-
Note 1: This short example doesn't illustrate - the fact that you can have more than a single injection point per - class. In such a case the names of the faults have to be different - if a developer wants to trigger them separately. -
-Note 2: After the injection step (see
- Putting It All Together)
- you can verify that the faults were properly injected by
- searching for ajc keywords in a disassembled class file.
-
For the sake of a unified naming - convention the following two types of names are recommended for a - new aspects development:
-fi.hdfs.DiskError
- fi.hdfs.datanode.BlockReceiver[optional location details]
- Faults (aspects) have to injected (or woven) together before - they can be used. Follow these instructions:
- -Faults can be triggered as follows: -
-
- As you can see above these two methods do the same thing. They are
- setting the probability level of hdfs.datanode.BlockReceiver
- at 12%. The difference, however, is that the program provides more
- flexibility and allows you to turn a fault off when a test no longer needs it.
-
These two sources of information are particularly - interesting and worth reading: -
-If you have additional comments or questions for the author check - HDFS-435. -
-- Offline Edits Viewer is a tool to parse the Edits log file. The - current processors are mostly useful for conversion between - different formats, including XML which is human readable and - easier to edit than native binary format. -
- -- The tool can parse the edits formats -18 (roughly Hadoop 0.19) - and later. The tool operates on files only, it does not need - Hadoop cluster to be running. -
- -Input formats supported:
-- The Offline Edits Viewer provides several output processors - (unless stated otherwise the output of the processor can be - converted back to original edits file): -
-bash$ bin/hdfs oev -i edits -o edits.xml
| Flag | Description |
|---|---|
[-i|--inputFile] <input file> |
- - Specify the input edits log file to process. Xml (case - insensitive) extension means XML format otherwise binary - format is assumed. Required. - | -
[-o|--outputFile] <output file> |
- - Specify the output filename, if the specified output processor - generates one. If the specified file already exists, it is - silently overwritten. Required. - | -
[-p|--processor] <processor> |
- - Specify the image processor to apply against the image - file. Currently valid options are binary, - xml (default) and stats. - | -
[-v|--verbose]- |
- - Print the input and output filenames and pipe output of - processor to console as well as specified file. On extremely - large files, this may increase processing time by an order - of magnitude. - | -
[-h|--help] |
- - Display the tool usage and help information and exit. - | -
- In case there is some problem with hadoop cluster and the edits - file is corrupted it is possible to save at least part of the - edits file that is correct. This can be done by converting the - binary edits to XML, edit it manually and then convert it back - to binary. The most common problem is that the edits file is - missing the closing record (record that has opCode -1). This - should be recognized by the tool and the XML format should be - properly closed. -
- -- If there is no closing record in the XML file you can add one - after last correct record. Anything after the record with opCode - -1 is ignored. -
- -Example of a closing record (with opCode -1):
-The Offline Image Viewer is a tool to dump the contents of hdfs - fsimage files to human-readable formats in order to allow offline analysis - and examination of an Hadoop cluster's namespace. The tool is able to - process very large image files relatively quickly, converting them to - one of several output formats. The tool handles the layout formats that - were included with Hadoop versions 16 and up. If the tool is not able to - process an image file, it will exit cleanly. The Offline Image Viewer does not require - an Hadoop cluster to be running; it is entirely offline in its operation.
- -The Offline Image Viewer provides several output processors:
-lsr command. It includes the same fields, in the same order, as
- lsr : directory or file flag, permissions, replication, owner, group,
- file size, modification date, and full path. Unlike the lsr command,
- the root path is included. One important difference between the output
- of the lsr command this processor, is that this output is not sorted
- by directory name and contents. Rather, the files are listed in the
- order in which they are stored in the fsimage file. Therefore, it is
- not possible to directly compare the output of the lsr command this
- this tool. The Ls processor uses information contained within the Inode blocks to
- calculate file sizes and ignores the -skipBlocks option.lsr format is suitable for easy human comprehension.-delimiter command line argument. This processor is designed to
- create output that is easily analyzed by other tools, such as Apache Pig.
- See the Analyzing Results section
- for further information on using this processor to analyze the contents of fsimage files.lsr processor. The output
- of this processor is amenable to automated processing and analysis with XML tools.
- Due to the verbosity of the XML syntax, this processor will also generate
- the largest amount of output.[0, maxSize] by specifying
- maxSize and a step.
- The range of integers is divided into segments of size
- step:
- [0, s1, ..., sn-1, maxSize],
- and the processor calculates how many files in the system fall into
- each segment [si-1, si).
- Note that files larger than maxSize always fall into
- the very last segment.
- The output file is formatted as a tab separated two column table:
- Size and NumFiles. Where Size represents the start of the segment,
- and numFiles is the number of files form the image which size falls
- in this segment.The simplest usage of the Offline Image Viewer is to provide just an input and output
- file, via the -i and -o command-line switches:
bash$ bin/hdfs oiv -i fsimage -o fsimage.txt
This will create a file named fsimage.txt in the current directory using - the Ls output processor. For very large image files, this process may take - several minutes.
- -One can specify which output processor via the command-line switch -p.
- For instance:
bash$ bin/hdfs oiv -i fsimage -o fsimage.xml -p XML
or
- -bash$ bin/hdfs oiv -i fsimage -o fsimage.txt -p Indented
This will run the tool using either the XML or Indented output processor, - respectively.
- -One command-line option worth considering is -skipBlocks, which
- prevents the tool from explicitly enumerating all of the blocks that make up
- a file in the namespace. This is useful for file systems that have very large
- files. Enabling this option can significantly decrease the size of the resulting
- output, as individual blocks are not included. Note, however, that the Ls processor
- needs to enumerate the blocks and so overrides this option.
Consider the following contrived namespace:
-Applying the Offline Image Processor against this file with default options would result in the following output:
-Similarly, applying the Indented processor would generate output that begins with:
-| Flag | Description |
|---|---|
[-i|--inputFile] <input file> |
- Specify the input fsimage file to process. Required. |
[-o|--outputFile] <output file> |
- Specify the output filename, if the specified output processor - generates one. If the specified file already exists, it is silently overwritten. Required. - |
[-p|--processor] <processor> |
- Specify the image processor to apply against the image file. Currently - valid options are Ls (default), XML and Indented.. - |
-skipBlocks |
- Do not enumerate individual blocks within files. This may save processing time
- and outfile file space on namespaces with very large files. The Ls processor reads
- the blocks to correctly determine file sizes and ignores this option. |
-printToScreen |
- Pipe output of processor to console as well as specified file. On extremely - large namespaces, this may increase processing time by an order of magnitude. |
-delimiter <arg> |
- When used in conjunction with the Delimited processor, replaces the default
- tab delimiter with the string specified by arg. |
[-h|--help] |
- Display the tool usage and help information and exit. |
The Offline Image Viewer makes it easy to gather large amounts of data about the hdfs namespace. - This information can then be used to explore file system usage patterns or find - specific files that match arbitrary criteria, along with other types of namespace analysis. The Delimited - image processor in particular creates - output that is amenable to further processing by tools such as Apache Pig. Pig provides a particularly - good choice for analyzing these data as it is able to deal with the output generated from a small fsimage - but also scales up to consume data from extremely large file systems.
-The Delimited image processor generates lines of text separated, by default, by tabs and includes - all of the fields that are common between constructed files and files that were still under constructed - when the fsimage was generated. Examples scripts are provided demonstrating how to use this output to - accomplish three tasks: determine the number of files each user has created on the file system, - find files were created but have not accessed, and find probable duplicates of large files by comparing - the size of each file.
-Each of the following scripts assumes you have generated an output file using the Delimited processor named
- foo and will be storing the results of the Pig analysis in a file named results.
This script processes each path within the namespace, groups them by the file owner and determines the total - number of files each user owns.
-numFilesOfEachUser.pig:
-This script can be run against pig with the following command:
-bin/pig -x local -param inputFile=../foo -param outputFile=../results ../numFilesOfEachUser.pig
The output file's content will be similar to that below:
-
- bart 1
- lisa 16
- homer 28
- marge 2456
-
This script finds files that were created but whose access times were never changed, meaning they were never opened or viewed.
-neverAccessed.pig:
-This script can be run against pig with the following command and its output file's content will be a list of files that were created but never viewed afterwards.
-bin/pig -x local -param inputFile=../foo -param outputFile=../results ../neverAccessed.pig
This script groups files together based on their size, drops any that are of less than 100mb and returns a list of the file size, number of files found and a tuple of the file paths. This can be used to find likely duplicates within the filesystem namespace.
- -probableDuplicates.pig:
-This script can be run against pig with the following command:
-bin/pig -x local -param inputFile=../foo -param outputFile=../results ../probableDuplicates.pig
The output file's content will be similar to that below:
- -Each line includes the file size in bytes that was found to be duplicated, the number of duplicates found, and a list of the duplicated paths. - Files less than 100MB are ignored, providing a reasonable likelihood that files of these exact sizes may be duplicates.
-- The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model. - Each file and directory is associated with an owner and a group. The file or directory has separate permissions for the - user that is the owner, for other users that are members of the group, and for all other users. - - For files, the r permission is required to read the file, and the w permission is required to write or append to the file. - - For directories, the r permission is required to list the contents of the directory, the w permission is required to create - or delete files or directories, and the x permission is required to access a child of the directory. -
-- In contrast to the POSIX model, there are no setuid or setgid bits for files as there is no notion of executable files. - For directories, there are no setuid or setgid bits directory as a simplification. The Sticky bit can be set - on directories, preventing anyone except the superuser, directory owner or file owner from deleting or moving the files within the directory. - Setting the sticky bit for a file has no effect. Collectively, the permissions of a file or directory are its mode. In general, Unix - customs for representing and displaying modes will be used, including the use of octal numbers in this description. When a file or directory - is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule). -
-
- Each client process that accesses HDFS has a two-part identity composed of the user name, and groups list.
- Whenever HDFS must do a permissions check for a file or directory foo accessed by a client process,
-
foo, then the owner permissions are tested;
- foo matches any of member of the groups list, then the group permissions are tested;
- foo are tested.
- - If a permissions check fails, the client operation fails. -
-
-As of Hadoop 0.22, Hadoop supports two different modes of operation to determine the user's identity, specified by the
-hadoop.security.authentication property:
-
simple`whoami`.kerberoskinit utility to obtain a Kerberos ticket-granting-ticket (TGT) and
- use klist to determine their current principal. When mapping a Kerberos principal to an HDFS username, all components except for the primary are dropped. For example, a principal todd/foobar@CORP.COMPANY.COM will act as the simple username todd on HDFS.
- -Regardless of the mode of operation, the user identity mechanism is extrinsic to HDFS itself. -There is no provision within HDFS for creating user identities, establishing groups, or processing user credentials. -
-
-Once a username has been determined as described above, the list of groups is
-determined by a group mapping service, configured by the
-hadoop.security.group.mapping property. Refer to the
-core-default.xml for details of the hadoop.security.group.mapping
-implementation.
-
-An alternate implementation, which connects directly to an LDAP server to resolve the list of groups, is available
-via org.apache.hadoop.security.LdapGroupsMapping. However, this provider should only be used if the
-required groups reside exclusively in LDAP, and are not materialized on the Unix servers. More information on
-configuring the group mapping service is available in the Javadocs.
-
-For HDFS, the mapping of users to groups is performed on the NameNode. Thus, the host system configuration of -the NameNode determines the group mappings for the users. -
--Note that HDFS stores the user and group of a file or directory as strings; there is no conversion from user and -group identity numbers as is conventional in Unix. -
- --Each file or directory operation passes the full path name to the name node, and the permissions checks are applied along the -path for each operation. The client framework will implicitly associate the user identity with the connection to the name node, -reducing the need for changes to the existing client API. It has always been the case that when one operation on a file succeeds, -the operation might fail when repeated because the file, or some directory on the path, no longer exists. For instance, when the -client first begins reading a file, it makes a first request to the name node to discover the location of the first blocks of the file. -A second request made to find additional blocks may fail. On the other hand, deleting a file does not revoke access by a client -that already knows the blocks of the file. With the addition of permissions, a client's access to a file may be withdrawn between -requests. Again, changing permissions does not revoke the access of a client that already knows the file's blocks. -
-
- All methods that use a path parameter will throw AccessControlException if permission checking fails.
-
New methods:
-public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short
- replication, long blockSize, Progressable progress) throws IOException;
- public boolean mkdirs(Path f, FsPermission permission) throws IOException;
- public void setPermission(Path p, FsPermission permission) throws IOException;
- public void setOwner(Path p, String username, String groupname) throws IOException;
- public FileStatus getFileStatus(Path f) throws IOException; will additionally return the user,
- group and mode associated with the path.
-
-The mode of a new file or directory is restricted my the umask set as a configuration parameter.
-When the existing create(path, …) method (without the permission parameter)
-is used, the mode of the new file is 666 & ^umask. When the
-new create(path, permission, …) method
-(with the permission parameter P) is used, the mode of the new file is
-P & ^umask & 666. When a new directory is
-created with the existing mkdirs(path) method (without the permission parameter),
-the mode of the new directory is 777 & ^umask. When the
-new mkdirs(path, permission ) method (with the
-permission parameter P) is used, the mode of new directory is
-P & ^umask & 777.
-
New operations:
-chmod [-R] mode file …
- chgrp [-R] group file …
- chgrp must belong to the specified group and be the owner of the file, or be the super-user.
- chown [-R] [owner][:[group]] file …
- ls file …
- lsr file …
- - The super-user is the user with the same identity as name node process itself. Loosely, if you started the name - node, then you are the super-user. The super-user can do anything in that permissions checks never fail for the - super-user. There is no persistent notion of who was the super-user; when the name node is started - the process identity determines who is the super-user for now. The HDFS super-user does not have - to be the super-user of the name node host, nor is it necessary that all clusters have the same super-user. Also, - an experimenter running HDFS on a personal workstation, conveniently becomes that installation's super-user - without any configuration. -
-- In addition, the administrator my identify a distinguished group using a configuration parameter. If set, members - of this group are also super-users. -
--By default, the identity of the web server is a configuration parameter. That is, the name node has no notion of the identity of -the real user, but the web server behaves as if it has the identity (user and groups) of a user chosen -by the administrator. Unless the chosen identity matches the super-user, parts of the name space may be inaccessible -to the web server.
-dfs.permissions = true
- yes use the permissions system as described here. If no, permission
- checking is turned off, but all other behavior is unchanged. Switching from one parameter
- value to the other does not change the mode, owner or group of files or directories.
- chmod, chgrp and
- chown always check permissions. These functions are only useful in the
- permissions context, and so there is no backwards compatibility issue. Furthermore, this allows
- administrators to reliably set owners and permissions in advance of turning on regular permissions checking.
- dfs.web.ugi = webuser,webgroup
- dfs.permissions.superusergroup = supergroup
- fs.permissions.umask-mode = 022
- umask used when creating files and directories. For configuration files, the decimal
- value 1810 may be used.
- dfs.cluster.administrators = ACL-for-admins>
- The Hadoop Distributed File System (HDFS) allows the administrator to set quotas for the number of names used and the -amount of space used for individual directories. Name quotas and space quotas operate independently, but the administration and -implementation of the two types of quotas are closely parallel.
- The name quota is a hard limit on the number of file and directory names in the tree rooted at that directory. File and
-directory creations fail if the quota would be exceeded. Quotas stick with renamed directories; the rename operation fails if
-operation would result in a quota violation. The attempt to set a quota will still succeed even if the directory would be in violation of the new
-quota. A newly created directory has no associated quota. The largest quota is Long.Max_Value. A quota of one
-forces a directory to remain empty. (Yes, a directory counts against its own quota!)
Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in
-violation of a quota (perhaps the fsimage was surreptitiously modified),
-a warning is printed for each of such violations. Setting or removing a quota creates a journal entry.
The space quota is a hard limit on the number of bytes used by files in the tree rooted at that directory. Block
-allocations fail if the quota would not allow a full block to be written. Each replica of a block counts against the quota. Quotas
-stick with renamed directories; the rename operation fails if the operation would result in a quota violation. A newly created directory has no associated quota.
-The largest quota is Long.Max_Value. A quota of zero still permits files to be created, but no blocks can be added to the files.
-Directories don't use host file system space and don't count against the space quota. The host file system space used to save
-the file meta data is not counted against the quota. Quotas are charged at the intended replication factor for the file;
-changing the replication factor for a file will credit or debit quotas.
Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in
-violation of a quota (perhaps the fsimage was surreptitiously modified), a warning is printed for
-each of such violations. Setting or removing a quota creates a journal entry.
Quotas are managed by a set of commands available only to the administrator.
- -dfsadmin -setQuota <N> <directory>...<directory> N for
-each directory. Best effort for each directory, with faults reported if N is not a positive long integer, the
-directory does not exist or it is a file, or the directory would immediately exceed the new quota. dfsadmin -clrQuota <directory>...<director>dfsadmin -setSpaceQuota <N> <directory>...<directory> N is
-neither zero nor a positive integer, the directory does not exist or it is a file, or the directory would immediately exceed
-the new quota. dfsadmin -clrSpaceQuota <directory>...<director> An an extension to the count command of the HDFS shell reports quota values and the current count of names and bytes in use.
fs -count -q <directory>...<directory>-q option, also report the name quota
-value set for each directory, the available name quota remaining, the space quota value set, and the available space quota
-remaining. If the directory does not have a quota set, the reported values are none and inf.
-
- - This document is a starting point for users working with - Hadoop Distributed File System (HDFS) either as a part of a Hadoop cluster - or as a stand-alone general purpose distributed file system. - While HDFS is designed to "just work" in many environments, a working - knowledge of HDFS helps greatly with configuration improvements and - diagnostics on a specific cluster. -
-- HDFS is the primary distributed storage used by Hadoop applications. A - HDFS cluster primarily consists of a NameNode that manages the - file system metadata and DataNodes that store the actual data. The - HDFS Architecture Guide describes HDFS in detail. This user guide primarily deals with - the interaction of users and administrators with HDFS clusters. - The HDFS architecture diagram depicts - basic interactions among NameNode, the DataNodes, and the clients. - Clients contact NameNode for file metadata or file modifications and perform - actual file I/O directly with the DataNodes. -
-- The following are some of the salient features that could be of - interest to many users. -
-
fsck: a utility to diagnose health of the file system, to
- find missing files or blocks.
- fetchdt: a utility to fetch DelegationToken and store it
- in a file on the local system.
- - The following documents describe how to install and set up a Hadoop cluster: -
-- The rest of this document assumes the user is able to set up and run a - HDFS with at least one DataNode. For the purpose of this document, - both the NameNode and DataNode could be running on the same physical - machine. -
- -
- NameNode and DataNode each run an internal web server in order to
- display basic information about the current status of the cluster.
- With the default configuration, the NameNode front page is at
- http://namenode-name:50070/.
- It lists the DataNodes in the cluster and basic statistics of the
- cluster. The web interface can also be used to browse the file
- system (using "Browse the file system" link on the NameNode front
- page).
-
- Hadoop includes various shell-like commands that directly
- interact with HDFS and other file systems that Hadoop supports.
- The command
- bin/hdfs dfs -help
- lists the commands supported by Hadoop
- shell. Furthermore, the command
- bin/hdfs dfs -help command-name
- displays more detailed help for a command. These commands support
- most of the normal files system operations like copying files,
- changing file permissions, etc. It also supports a few HDFS
- specific operations like changing replication of files.
- For more information see File System Shell Guide.
-
- The bin/hadoop dfsadmin
- command supports a few HDFS administration related operations.
- The bin/hadoop dfsadmin -help command
- lists all the commands currently supported. For e.g.:
-
-report
- : reports basic statistics of HDFS. Some of this information is
- also available on the NameNode front page.
- -safemode
- : though usually not required, an administrator can manually enter
- or leave Safemode.
- -finalizeUpgrade
- : removes previous backup of the cluster made during last upgrade.
- -refreshNodes
- : Updates the namenode with the set of datanodes allowed to
- connect to the namenode. Namenodes re-read datanode hostnames
- in the file defined by dfs.hosts, dfs.hosts.exclude. Hosts defined
- in dfs.hosts are the datanodes that are part of the cluster.
- If there are entries in dfs.hosts, only the hosts in it are
- allowed to register with the namenode. Entries in dfs.hosts.exclude
- are datanodes that need to be decommissioned. Datanodes complete
- decommissioning when all the replicas from them are replicated
- to other datanodes. Decommissioned nodes are not automatically
- shutdown and are not chosen for writing for new replicas.
- -printTopology
- : Print the topology of the cluster. Display a tree of racks and
- datanodes attached to the tracks as viewed by the NameNode.
- - For command usage, see - dfsadmin. -
-
- The NameNode stores modifications to the file system as a log
- appended to a native file system file, edits.
- When a NameNode starts up, it reads HDFS state from an image
- file, fsimage, and then applies edits from the
- edits log file. It then writes new HDFS state to the fsimage
- and starts normal
- operation with an empty edits file. Since NameNode merges
- fsimage and edits files only during start up,
- the edits log file could get very large over time on a busy cluster.
- Another side effect of a larger edits file is that next
- restart of NameNode takes longer.
-
- The secondary NameNode merges the fsimage and the edits log files periodically - and keeps edits log size within a limit. It is usually run on a - different machine than the primary NameNode since its memory requirements - are on the same order as the primary NameNode. -
-- The start of the checkpoint process on the secondary NameNode is - controlled by two configuration parameters. -
-dfs.namenode.checkpoint.period, set to 1 hour by default, specifies
- the maximum delay between two consecutive checkpoints, and
- dfs.namenode.checkpoint.txns, set to 40000 default, defines the
- number of uncheckpointed transactions on the NameNode which will force
- an urgent checkpoint, even if the checkpoint period has not been reached.
- - The secondary NameNode stores the latest checkpoint in a - directory which is structured the same way as the primary NameNode's - directory. So that the check pointed image is always ready to be - read by the primary NameNode if necessary. -
-- For command usage, see - secondarynamenode. -
- -NameNode persists its namespace using two files: fsimage,
- which is the latest checkpoint of the namespace and edits,
- a journal (log) of changes to the namespace since the checkpoint.
- When a NameNode starts up, it merges the fsimage and
- edits journal to provide an up-to-date view of the
- file system metadata.
- The NameNode then overwrites fsimage with the new HDFS state
- and begins a new edits journal.
-
- The Checkpoint node periodically creates checkpoints of the namespace.
- It downloads fsimage and edits from the active
- NameNode, merges them locally, and uploads the new image back to the
- active NameNode.
- The Checkpoint node usually runs on a different machine than the NameNode
- since its memory requirements are on the same order as the NameNode.
- The Checkpoint node is started by
- bin/hdfs namenode -checkpoint on the node
- specified in the configuration file.
-
The location of the Checkpoint (or Backup) node and its accompanying
- web interface are configured via the dfs.namenode.backup.address
- and dfs.namenode.backup.http-address configuration variables.
-
- The start of the checkpoint process on the Checkpoint node is - controlled by two configuration parameters. -
-dfs.namenode.checkpoint.period, set to 1 hour by default, specifies
- the maximum delay between two consecutive checkpoints
- dfs.namenode.checkpoint.txns, set to 40000 default, defines the
- number of uncheckpointed transactions on the NameNode which will force
- an urgent checkpoint, even if the checkpoint period has not been reached.
- - The Checkpoint node stores the latest checkpoint in a - directory that is structured the same as the NameNode's - directory. This allows the checkpointed image to be always available for - reading by the NameNode if necessary. - See Import checkpoint. -
-Multiple checkpoint nodes may be specified in the cluster configuration file.
-- For command usage, see - namenode. -
-- The Backup node provides the same checkpointing functionality as the - Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the - file system namespace that is always synchronized with the active NameNode state. - Along with accepting a journal stream of file system edits from - the NameNode and persisting this to disk, the Backup node also applies - those edits into its own copy of the namespace in memory, thus creating - a backup of the namespace. -
-
- The Backup node does not need to download
- fsimage and edits files from the active NameNode
- in order to create a checkpoint, as would be required with a
- Checkpoint node or Secondary NameNode, since it already has an up-to-date
- state of the namespace state in memory.
- The Backup node checkpoint process is more efficient as it only needs to
- save the namespace into the local fsimage file and reset
- edits.
-
- As the Backup node maintains a copy of the - namespace in memory, its RAM requirements are the same as the NameNode. -
-- The NameNode supports one Backup node at a time. No Checkpoint nodes may be - registered if a Backup node is in use. Using multiple Backup nodes - concurrently will be supported in the future. -
-
- The Backup node is configured in the same manner as the Checkpoint node.
- It is started with bin/hdfs namenode -backup.
-
The location of the Backup (or Checkpoint) node and its accompanying
- web interface are configured via the dfs.namenode.backup.address
- and dfs.namenode.backup.http-address configuration variables.
-
- Use of a Backup node provides the option of running the NameNode with no
- persistent storage, delegating all responsibility for persisting the state
- of the namespace to the Backup node.
- To do this, start the NameNode with the
- -importCheckpoint option, along with specifying no persistent
- storage directories of type edits dfs.namenode.edits.dir
- for the NameNode configuration.
-
- For a complete discussion of the motivation behind the creation of the - Backup node and Checkpoint node, see - HADOOP-4539. - For command usage, see - namenode. -
-- The latest checkpoint can be imported to the NameNode if - all other copies of the image and the edits files are lost. - In order to do that one should: -
-dfs.namenode.name.dir configuration variable;
- dfs.namenode.checkpoint.dir;
- -importCheckpoint option.
-
- The NameNode will upload the checkpoint from the
- dfs.namenode.checkpoint.dir directory and then save it to the NameNode
- directory(s) set in dfs.namenode.name.dir.
- The NameNode will fail if a legal image is contained in
- dfs.namenode.name.dir.
- The NameNode verifies that the image in dfs.namenode.checkpoint.dir is
- consistent, but does not modify it in any way.
-
- For command usage, see - namenode. -
-- HDFS data might not always be be placed uniformly across the - DataNode. One common reason is addition of new DataNodes to an - existing cluster. While placing new blocks (data for a file is - stored as a series of blocks), NameNode considers various - parameters before choosing the DataNodes to receive these blocks. - Some of the considerations are: -
-- Due to multiple competing considerations, data might not be - uniformly placed across the DataNodes. - HDFS provides a tool for administrators that analyzes block - placement and rebalanaces data across the DataNode. A brief - administrator's guide for rebalancer as a - PDF - is attached to - HADOOP-1652. -
-- For command usage, see - balancer. -
- -
- Typically large Hadoop clusters are arranged in racks and
- network traffic between different nodes with in the same rack is
- much more desirable than network traffic across the racks. In
- addition NameNode tries to place replicas of block on
- multiple racks for improved fault tolerance. Hadoop lets the
- cluster administrators decide which rack a node belongs to
- through configuration variable net.topology.script.file.name. When this
- script is configured, each node runs the script to determine its
- rack id. A default installation assumes all the nodes belong to
- the same rack. This feature and configuration is further described
- in PDF
- attached to
- HADOOP-692.
-
- During start up the NameNode loads the file system state from the
- fsimage and the edits log file. It then waits for DataNodes
- to report their blocks so that it does not prematurely start
- replicating the blocks though enough replicas already exist in the
- cluster. During this time NameNode stays in Safemode.
- Safemode
- for the NameNode is essentially a read-only mode for the HDFS cluster,
- where it does not allow any modifications to file system or blocks.
- Normally the NameNode leaves Safemode automatically after the DataNodes
- have reported that most file system blocks are available.
- If required, HDFS could be placed in Safemode explicitly
- using 'bin/hadoop dfsadmin -safemode' command. NameNode front
- page shows whether Safemode is on or off. A more detailed
- description and configuration is maintained as JavaDoc for
- setSafeMode().
-
- HDFS supports the fsck command to check for various
- inconsistencies.
- It it is designed for reporting problems with various
- files, for example, missing blocks for a file or under-replicated
- blocks. Unlike a traditional fsck utility for native file systems,
- this command does not correct the errors it detects. Normally NameNode
- automatically corrects most of the recoverable failures. By default
- fsck ignores open files but provides an option to select all files during reporting.
- The HDFS fsck command is not a
- Hadoop shell command. It can be run as 'bin/hadoop fsck'.
- For command usage, see
- fsck.
- fsck can be run on the whole file system or on a subset of files.
-
- HDFS supports the fetchdt command to fetch Delegation Token
- and store it in a file on the local system. This token can be later used to
- access secure server (NameNode for example) from a non secure client.
- Utility uses either RPC or HTTPS (over Kerberos) to get the token, and thus
- requires kerberos tickets to be present before the run (run kinit to get
- the tickets).
- The HDFS fetchdt command is not a
- Hadoop shell command. It can be run as 'bin/hadoop fetchdt DTfile '.
- After you got the token you can run an HDFS command without having Kerberos
- tickets, by pointing HADOOP_TOKEN_FILE_LOCATION environmental variable to
- the delegation token file.
- For command usage, see fetchdt command.
-
Typically, you will configure multiple metadata storage locations. - Then, if one storage location is corrupt, you can read the - metadata from one of the other storage locations.
- -However, what can you do if the only storage locations available are - corrupt? In this case, there is a special NameNode startup mode called - Recovery mode that may allow you to recover most of your data.
- -You can start the NameNode in recovery mode like so:
- namenode -recover
When in recovery mode, the NameNode will interactively prompt you at - the command line about possible courses of action you can take to - recover your data.
- -If you don't want to be prompted, you can give the
- -force option. This option will force
- recovery mode to always select the first choice. Normally, this
- will be the most reasonable choice.
Because Recovery mode can cause you to lose data, you should always - back up your edit log and fsimage before using it.
-
- When Hadoop is upgraded on an existing cluster, as with any
- software upgrade, it is possible there are new bugs or
- incompatible changes that affect existing applications and were
- not discovered earlier. In any non-trivial HDFS installation, it
- is not an option to loose any data, let alone to restart HDFS from
- scratch. HDFS allows administrators to go back to earlier version
- of Hadoop and rollback the cluster to the state it was in
- before
- the upgrade. HDFS upgrade is described in more detail in
- Hadoop Upgrade Wiki page.
- HDFS can have one such backup at a time. Before upgrading,
- administrators need to remove existing backup using bin/hadoop
- dfsadmin -finalizeUpgrade command. The following
- briefly describes the typical upgrade procedure:
-
dfsadmin -upgradeProgress status
- can tell if the cluster needs to be finalized.
- -upgrade option
- (bin/start-dfs.sh -upgrade).
- bin/start-dfs.h -rollback).
- - The file permissions are designed to be similar to file permissions on - other familiar platforms like Linux. Currently, security is limited - to simple file permissions. The user that starts NameNode is - treated as the superuser for HDFS. Future versions of HDFS will - support network authentication protocols like Kerberos for user - authentication and encryption of data transfers. The details are discussed in the - Permissions Guide. -
- -- Hadoop currently runs on clusters with thousands of nodes. The - PoweredBy Wiki page - lists some of the organizations that deploy Hadoop on large - clusters. HDFS has one NameNode for each cluster. Currently - the total memory available on NameNode is the primary scalability - limitation. On very large clusters, increasing average size of - files stored in HDFS helps with increasing cluster size without - increasing memory requirements on NameNode. - - The default configuration may not suite very large clustes. The - FAQ Wiki page lists - suggested configuration improvements for large Hadoop clusters. -
- -- This user guide is a good starting point for - working with HDFS. While the user guide continues to improve, - there is a large wealth of documentation about Hadoop and HDFS. - The following list is a starting point for further exploration: -
-src/hdfs/hdfs-default.xml.
- It includes brief
- description of most of the configuration variables available.
- HFTP is a Hadoop filesystem implementation that lets you read data from a remote Hadoop HDFS cluster. - The reads are done via HTTP, and data is sourced from DataNodes. - HFTP is a read-only filesystem, and will throw exceptions if you try to use it to write data or modify - the filesystem state.
- -HFTP is primarily useful if you have multiple HDFS clusters with different versions and you need to move data from one to another. HFTP is wire-compatible even between different versions of HDFS. For example, you can do things like:
- hadoop distcp -i hftp://sourceFS:50070/src hdfs://destFS:50070/dest. Note that HFTP is read-only so the destination must be an HDFS filesystem. (Also, in this example, the distcp should be run using the configuraton of the new filesystem.)
An extension, HSFTP, uses HTTPS by default. This means that data will be encrypted in transit.
-The code for HFTP lives in the Java class org.apache.hadoop.hdfs.HftpFileSystem. Likewise,
- HSFTP is implemented in org.apache.hadoop.hdfs.HsftpFileSystem.
-
| Name | -Description | -
|---|---|
| dfs.hftp.https.port | -the HTTPS port on the remote cluster. If not set, HFTP will fall back on
- dfs.https.port. |
-
| hdfs.service.host_ip:port | -Specifies the service name (for the security subsystem) associated with the HFTP filesystem - running at ip:port. | -
-libhdfs is a JNI based C API for Hadoop's Distributed File System (HDFS). -It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and -the filesystem. libhdfs is part of the Hadoop distribution and comes -pre-compiled in ${HADOOP_PREFIX}/libhdfs/libhdfs.so . -
- --The libhdfs APIs are a subset of: hadoop fs APIs. -
--The header file for libhdfs describes each API in detail and is available in ${HADOOP_PREFIX}/src/c++/libhdfs/hdfs.h -
-
-See the Makefile for hdfs_test.c in the libhdfs source directory (${HADOOP_PREFIX}/src/c++/libhdfs/Makefile) or something like:
-gcc above_sample.c -I${HADOOP_PREFIX}/src/c++/libhdfs -L${HADOOP_PREFIX}/libhdfs -lhdfs -o above_sample
-
-The most common problem is the CLASSPATH is not set properly when calling a program that uses libhdfs. -Make sure you set it to all the Hadoop jars needed to run Hadoop itself. Currently, there is no way to -programmatically generate the classpath, but a good bet is to include all the jar files in ${HADOOP_PREFIX} -and ${HADOOP_PREFIX}/lib as well as the right configuration directory containing hdfs-site.xml -
-libdhfs is thread safe.
-