- The Hadoop Distributed File System (HDFS) implements a permissions model for files and directories that shares much of the POSIX model. Each file and directory is associated with an owner and a group. The file or directory has separate permissions for the user that is the owner, for other users that are members of the group, and for all other users. For files, the r permission is required to read the file, and the w permission is required to write or append to the file. For directories, the r permission is required to list the contents of the directory, the w permission is required to create or delete files or directories, and the x permission is required to access a child of the directory. In contrast to the POSIX model, there are no setuid or setgid bits for files as there is no notion of executable files. For directories, there are no setuid or setgid bits directory as a simplification. The Sticky bit can be set on directories, preventing anyone except the superuser, directory owner or file owner from deleting or moving the files within the directory. Setting the sticky bit for a file has no effect. Collectively, the permissions of a file or directory are its mode. In general, Unix customs for representing and displaying modes will be used, including the use of octal numbers in this description. When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule). -
-
- Each client process that accesses HDFS has a two-part identity composed of the user name, and groups list. Whenever HDFS must do a permissions check for a file or directory foo accessed by a client process,
-
foo, then the owner permissions are tested;
- foo matches any of member of the groups list, then the group permissions are tested;
- foo are tested.
- - If a permissions check fails, the client operation fails. -
--In this release of Hadoop the identity of a client process is just whatever the host operating system says it is. For Unix-like systems, -
-`whoami`;
-`bash -c groups`.
--In the future there will be other ways of establishing user identity (think Kerberos, LDAP, and others). There is no expectation that this first method is secure in protecting one user from impersonating another. This user identity mechanism combined with the permissions model allows a cooperative community to share file system resources in an organized fashion. -
--In any case, the user identity mechanism is extrinsic to HDFS itself. There is no provision within HDFS for creating user identities, establishing groups, or processing user credentials. -
--Each file or directory operation passes the full path name to the name node, and the permissions checks are applied along the path for each operation. The client framework will implicitly associate the user identity with the connection to the name node, reducing the need for changes to the existing client API. It has always been the case that when one operation on a file succeeds, the operation might fail when repeated because the file, or some directory on the path, no longer exists. For instance, when the client first begins reading a file, it makes a first request to the name node to discover the location of the first blocks of the file. A second request made to find additional blocks may fail. On the other hand, deleting a file does not revoke access by a client that already knows the blocks of the file. With the addition of permissions, a client's access to a file may be withdrawn between requests. Again, changing permissions does not revoke the access of a client that already knows the file's blocks. -
--The map-reduce framework delegates the user identity by passing strings without special concern for confidentiality. The owner and group of a file or directory are stored as strings; there is no conversion from user and group identity numbers as is conventional in Unix. -
--The permissions features of this release did not require any changes to the behavior of data nodes. Blocks on the data nodes do not have any of the Hadoop ownership or permissions attributes associated with them. -
-
- All methods that use a path parameter will throw AccessControlException if permission checking fails.
-
New methods:
-public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException;
- public boolean mkdirs(Path f, FsPermission permission) throws IOException;
- public void setPermission(Path p, FsPermission permission) throws IOException;
- public void setOwner(Path p, String username, String groupname) throws IOException;
- public FileStatus getFileStatus(Path f) throws IOException; will additionally return the user, group and mode associated with the path.
-
-The mode of a new file or directory is restricted my the umask set as a configuration parameter. When the existing create(path, …) method (without the permission parameter) is used, the mode of the new file is 666 & ^umask. When the new create(path, permission, …) method (with the permission parameter P) is used, the mode of the new file is P & ^umask & 666. When a new directory is created with the existing mkdirs(path) method (without the permission parameter), the mode of the new directory is 777 & ^umask. When the new mkdirs(path, permission ) method (with the permission parameter P) is used, the mode of new directory is P & ^umask & 777.
-
New operations:
-chmod [-R] mode file …chgrp [-R] group file …chgrp must belong to the specified group and be the owner of the file, or be the super-user.
- chown [-R] [owner][:[group]] file …ls file …lsr file …- The super-user is the user with the same identity as name node process itself. Loosely, if you started the name node, then you are the super-user. The super-user can do anything in that permissions checks never fail for the super-user. There is no persistent notion of who was the super-user; when the name node is started the process identity determines who is the super-user for now. The HDFS super-user does not have to be the super-user of the name node host, nor is it necessary that all clusters have the same super-user. Also, an experimenter running HDFS on a personal workstation, conveniently becomes that installation's super-user without any configuration. -
-- In addition, the administrator my identify a distinguished group using a configuration parameter. If set, members of this group are also super-users. -
--The identity of the web server is a configuration parameter. That is, the name node has no notion of the identity of the real user, but the web server behaves as if it has the identity (user and groups) of a user chosen by the administrator. Unless the chosen identity matches the super-user, parts of the name space may be invisible to the web server.
-
-If a cluster starts with a version 0.15 data set (fsimage), all files and directories will have owner O, group G, and mode M, where O and G are the user and group identity of the super-user, and M is a configuration parameter.
dfs.permissions = true yes use the permissions system as described here. If no, permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories.
- -
- Regardless of whether permissions are on or off,chmod, chgrp and chown always check permissions. These functions are only useful in the permissions context, and so there is no backwards compatibility issue. Furthermore, this allows administrators to reliably set owners and permissions in advance of turning on regular permissions checking.
- dfs.web.ugi = webuser,webgroupdfs.permissions.supergroup = supergroupdfs.upgrade.permission = 0777dfs.umaskmode = 022umask used when creating files and directories. May be specified either via three octal digits or symbolic values, with the same constraints as the dfs chmod command.
- -libhdfs is a JNI based C api for Hadoop's DFS. It provides C apis to a subset of the HDFS APIs to manipulate DFS files and the filesystem. libhdfs is part of the hadoop distribution and comes pre-compiled in ${HADOOP_HOME}/libhdfs/libhdfs.so . -
- --The libhdfs APIs are a subset of: hadoop fs APIs. -
--The header file for libhdfs describes each API in detail and is available in ${HADOOP_HOME}/src/c++/libhdfs/hdfs.h -
--See the Makefile for hdfs_test.c in the libhdfs source directory (${HADOOP_HOME}/src/c++/libhdfs/Makefile) or something like: -gcc above_sample.c -I${HADOOP_HOME}/src/c++/libhdfs -L${HADOOP_HOME}/libhdfs -lhdfs -o above_sample -
--The most common problem is the CLASSPATH is not set properly when calling a program that uses libhdfs. Make sure you set it to all the hadoop jars needed to run Hadoop itself. Currently, there is no way to programmatically generate the classpath, but a good bet is to include all the jar files in ${HADOOP_HOME} and ${HADOOP_HOME}/lib as well as the right configuration directory containing hdfs-site.xml -
-Concurrency and Hadoop FS "handles" - the hadoop FS implementation includes a FS handle cache which caches based on the URI of the namenode along with the user connecting. So, all calls to hdfsConnect will return the same handle but calls to hdfsConnectAsUser with different users will return different handles. But, since HDFS client handles are completely thread safe, this has no bearing on concurrency. -
-Concurrency and libhdfs/JNI - the libhdfs calls to JNI should always be creating thread local storage, so (in theory), libhdfs should be as thread safe as the underlying calls to the Hadoop FS. -
--Hadoop streaming is a utility that comes with the Hadoop distribution. The utility allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer. For example: -
--In the above example, both the mapper and the reducer are executables that read the input from stdin (line by line) and emit the output to stdout. The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes. -
- When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the - prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) will be the value. - If there is no tab character in the line, then entire line is considered as key and the value is null. However, this can be customized, as discussed later. -
--When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. However, this can be customized, as discussed later. -
-This is the basis for the communication protocol between the Map/Reduce framework and the streaming mapper/reducer. -
-You can supply a Java class as the mapper and/or the reducer. The above example is equivalent to: -
-User can specify stream.non.zero.exit.is.failure as
-true or false to make a streaming task that exits
-with a non-zero status to be Failure
-or Success respectively. By default, streaming tasks exiting
-with non-zero status are considered to be failed tasks.
-You can specify any executable as the mapper and/or the reducer. The executables do not need to pre-exist on the machines in the cluster; however, if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission. For example: -
--The above example specifies a user defined Python executable as the mapper. The option "-file myPythonScript.py" causes the python executable shipped to the cluster machines as a part of job submission. -
--In addition to executable files, you can also package other auxiliary files (such as dictionaries, configuration files, etc) that may be used by the mapper and/or the reducer. For example: -
--Often, you may want to process input data using a map function only. To do this, simply set mapred.reduce.tasks to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job. -
-To be backward compatible, Hadoop Streaming also supports the "-reduce NONE" option, which is equivalent to "-D mapred.reduce.tasks=0". -
--Just as with a normal Map/Reduce job, you can specify other plugins for a streaming job: -
--The class you supply for the input format should return key/value pairs of Text class. If you do not specify an input format class, the TextInputFormat is used as the default. Since the TextInputFormat returns keys of LongWritable class, which are actually not part of the input data, the keys will be discarded; only the values will be piped to the streaming mapper. -
-The class you supply for the output format is expected to take key/value pairs of Text class. If you do not specify an output format class, the TextOutputFormat is used as the default. -
--The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs_port values from the fs.default.name config variable. -
--Here are examples of the -files option: -
--In the above example, the part of the url after # is used as the symlink name that is created in the current working directory of tasks. So the tasks will have a symlink called testlink in the cwd that points to a local copy of testfile.txt. Multiple entries can be specified as: -
--The -archives option allows you to copy jars locally to the cwd of tasks and automatically unjar the files. For example: -
--In the example above, a symlink testlink3 is created in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file. -
--Here's another example of the -archives option. Here, the input.txt file has two lines specifying the names of the two files: testlink/cache.txt and testlink/cache2.txt. "testlink" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt". -
--You can specify additional configuration variables by using "-D <n>=<v>". For example: -
--The -D mapred.reduce.tasks=2 in the above example specifies to use two reducers for the job. -
--For more details on the jobconf parameters see: -mapred-default.html
--Other options you may specify for a streaming job are described here: -
-| Parameter | Optional/Required | Description |
|---|---|---|
| -cmdenv name=value | Optional | Pass env var to streaming commands |
| -inputreader JavaClassName | Optional | For backwards-compatibility: specifies a record reader class (instead of an input format class) |
| -verbose | Optional | Verbose output |
| -lazyOutput | Optional | Create output lazily. For example, if the output format is based on FileOutputFormat, the output file is created only on the first call to output.collect (or Context.write) |
-Streaming support Hadoop generic command line options.
-
-Supported parameters are :
-The general command line syntax is :
-
bin/hadoop command [genericOptions] [commandOptions]
-
| Parameter | Optional/Required | Description |
|---|---|---|
| -conf configuration_file | Optional | specify an application configuration file |
| -D property=value | Optional | use value for given property |
| -fs host:port or local | Optional | specify a namenode |
| -jt host:port or local | Optional | specify a job tracker |
| -files | Optional | specify comma separated files to be copied to the map reduce cluster |
| -archives | Optional | specify comma separated archives to be unarchived on the compute machines |
| Optional | ||
| -jt host:port or local | Optional |
-To change the local temp directory use: -
--To specify additional local temp directories use: -
--For more details on jobconf parameters see: -mapred-default.html
--To set an environment variable in a streaming command use: -
--As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the rest of the line (excluding the tab character) is the value. -
--However, you can customize this default. You can specify a field separator other than the tab character (the default), and you can specify the nth (n >= 1) character rather than the first character in a line (the default) as the separator between the key and value. For example: -
- --In the above example, "-D stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")). -
-Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value. -
-Similarly, you can specify "stream.map.input.field.separator" and -"stream.reduce.input.field.separator" as the input separator for map/reduce -inputs. By default the separator is the tab character.
--Hadoop has a library class, -KeyFieldBasedPartitioner, -that is useful for many applications. This class allows the Map/Reduce -framework to partition the map outputs based on certain key fields, not -the whole keys. For example: -
--Here, -D stream.map.output.field.separator=. and -D stream.num.map.output.key.fields=4 are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper. -
-The map output keys of the above Map/Reduce job normally have four fields -separated by ".". However, the Map/Reduce framework will partition the map -outputs by the first two fields of the keys using the --D mapred.text.key.partitioner.options=-k1,2 option. -Here, -D map.output.key.field.separator=. specifies the separator -for the partition. This guarantees that all the key/value pairs with the -same first two fields in the keys will be partitioned into the same reducer. -
-This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting. A simple illustration is shown here: -
--Output of map (the keys)
-Partition into 3 reducers (the first 2 fields are used as keys for partition)
-Sorting within each partition for the reducer(all 4 fields used for sorting)
-Hadoop has a library class, -KeyFieldBasedComparator, -that is useful for many applications. This class provides a subset of features -provided by the Unix/GNU Sort. For example: -
--The map output keys of the above Map/Reduce job normally have four fields -separated by ".". However, the Map/Reduce framework will sort the -outputs by the second field of the keys using the --D mapred.text.key.comparator.options=-k2,2nr option. -Here, -n specifies that the sorting is numerical sorting and --r specifies that the result should be reversed. A simple illustration -is shown below: -
--Output of map (the keys)
--Sorting output for the reducer(where second field used for sorting)
--Hadoop has a library package called -Aggregate. -Aggregate provides a special reducer class and a special combiner class, and -a list of simple aggregators that perform aggregations such as "sum", "max", -"min" and so on over a sequence of values. Aggregate allows you to define a -mapper plugin class that is expected to generate "aggregatable items" for each -input key/value pair of the mappers. The combiner/reducer will aggregate those -aggregatable items by invoking the appropriate aggregators. -
-To use Aggregate, simply specify "-reducer aggregate": -
--The python program myAggregatorForKeyCount.py looks like: -
--Hadoop has a library class, org.apache.hadoop.mapred.lib.FieldSelectionMapReduce, that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example: -
--The option "-D map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all -the subsequent fields). -
-The option "-D reduce.output.key.value.fields.spec=0-2:5-" specifies -key/value selection for the reduce outputs. In this case, the reduce -output key will consist of fields 0, 1, 2 (corresponding to the original -fields 6, 5, 1). The reduce output value will consist of all fields starting -from field 5 (corresponding to all the original fields). -
--Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this. -
- --As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this using either of these methods: -
-See the Hadoop Wiki for details: Reducer -
--Using an alias will not work, but variable substitution is allowed as shown in this example: -
--Currently this does not work and gives an "java.io.IOException: Broken pipe" error. This is probably a bug that needs to be investigated. -
--The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space: -
--You can specify multiple input directories with multiple '-input' options: -
-Instead of plain text files, you can generate gzip files as your generated output. Pass '-D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec' as option to your streaming job. -
--At least as late as version 0.14, Hadoop does not support multiple jar files. So, when specifying your own custom classes you will have to pack them along with the streaming jar and use the custom jar instead of the default hadoop streaming jar. -
--You can use the record reader StreamXmlRecordReader to process XML documents. -
--Anything found between BEGIN_STRING and END_STRING would be treated as one record for map tasks. -
-
-A streaming process can use the stderr to emit counter information.
-reporter:counter:<group>,<counter>,<amount>
-should be sent to stderr to update the counter.
-
-A streaming process can use the stderr to emit status information.
-To set a status, reporter:status:<message> should be sent
-to stderr.
-