Applications specify the files, via urls (hdfs:// or http://) to be cached via the {@link JobConf}. The DistributedCache assumes that the files specified via hdfs:// urls are already present on the {@link FileSystem} at the path specified by the url.

The framework will copy the necessary files on to the slave node before any tasks for the job are executed on that node. Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves.

DistributedCache can be used to distribute simple, read-only data/text files and/or more complex types such as archives, jars etc. Archives (zip, tar and tgz/tar.gz files) are un-archived at the slave nodes. Jars may be optionally added to the classpath of the tasks, a rudimentary software distribution mechanism. Files have execution permissions. Optionally users can also direct it to symlink the distributed cache file(s) into the working directory of the task.

DistributedCache tracks modification timestamps of the cache files. Clearly the cache files should not be modified by the application or externally while the job is executing.

Here is an illustrative example on how to use the DistributedCache:

     // Setting up the cache for the application
     
     1. Copy the requisite files to the FileSystem:
     
     $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat  
     $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip  
     $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar
     $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar
     $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz
     $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz
     
     2. Setup the application's JobConf:
     
     JobConf job = new JobConf();
     DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"), 
                                   job);
     DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
     DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
     DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
     
     3. Use the cached files in the {@link Mapper} or {@link Reducer}:
     
     public static class MapClass extends MapReduceBase  
     implements Mapper<K, V, K, V> {
     
       private Path[] localArchives;
       private Path[] localFiles;
       
       public void configure(JobConf job) {
         // Get the cached archives/files
         localArchives = DistributedCache.getLocalCacheArchives(job);
         localFiles = DistributedCache.getLocalCacheFiles(job);
       }
       
       public void map(K key, V value, 
                       OutputCollector<K, V> output, Reporter reporter) 
       throws IOException {
         // Use data from the cached archives/files here
         // ...
         // ...
         output.collect(k, v);
       }
     }
     
 

Typical usage is something like the following:

 DataInputBuffer buffer = new DataInputBuffer();
 while (... loop condition ...) {
   byte[] data = ... get data ...;
   int dataLength = ... get data length ...;
   buffer.reset(data, dataLength);
   ... read buffer using DataInput methods ...
 }
 

Typical usage is something like the following:

 DataOutputBuffer buffer = new DataOutputBuffer();
 while (... loop condition ...) {
   buffer.reset();
   ... write buffer using DataOutput methods ...
   byte[] data = buffer.getData();
   int dataLength = buffer.getLength();
   ... write data to its ultimate destination ...
 }
 

Typical usage is something like the following:

 InputBuffer buffer = new InputBuffer();
 while (... loop condition ...) {
   byte[] data = ... get data ...;
   int dataLength = ... get data length ...;
   buffer.reset(data, dataLength);
   ... read buffer using InputStream methods ...
 }
 

The index file is read entirely into memory. Thus key implementations should try to keep themselves small.

Map files are created by adding entries in-order. To maintain a large database, perform updates by copying the previous version of a database and merging in a sorted change list, to create a new version of the database in a new file. Sorting large change lists can be done with {@link SequenceFile.Sorter}.]]> key and val. Returns true if such a pair exists and false when at the end of the map]]> key or if it does not exist, at the first entry after the named key. - * @param key - key that we're trying to find - * @param val - data value if key is found - * @return - the key that was the closest match or null if eof.]]> key does not exist, return the first entry that falls just before the key. Otherwise, return the record that sorts just after. @return - the key that was the closest match or null if eof.]]> o is an MD5Hash whose digest contains the same values.]]> This saves memory over creating a new OutputStream and ByteArrayOutputStream each time data is written.

Typical usage is something like the following:

 OutputBuffer buffer = new OutputBuffer();
 while (... loop condition ...) {
   buffer.reset();
   ... write buffer using OutputStream methods ...
   byte[] data = buffer.getData();
   int dataLength = buffer.getLength();
   ... write data to its ultimate destination ...
 }
 

SequenceFile provides {@link Writer}, {@link Reader} and {@link Sorter} classes for writing, reading and sorting respectively.

1. Writer : Uncompressed records.
2. RecordCompressWriter : Record-compressed files, only compress values.
3. BlockCompressWriter : Block-compressed files, both keys & values are collected in 'blocks' separately and compressed. The size of the 'block' is configurable.

The actual compression algorithm used to compress key and/or values can be specified by using the appropriate {@link CompressionCodec}.

The recommended way is to use the static createWriter methods provided by the SequenceFile to chose the preferred format.

The {@link Reader} acts as the bridge and can read any of the above SequenceFile formats.

SequenceFile Formats

Essentially there are 3 different formats for SequenceFiles depending on the CompressionType specified. All of them share a common header described below.

SequenceFile Header

• version - 3 bytes of magic header SEQ, followed by 1 byte of actual version number (e.g. SEQ4 or SEQ6)
• keyClassName -key class
• valueClassName - value class
• compression - A boolean which specifies if compression is turned on for keys/values in this file.
• blockCompression - A boolean which specifies if block-compression is turned on for keys/values in this file.
• compression codec - CompressionCodec class which is used for compression of keys and/or values (if compression is enabled).
• metadata - {@link Metadata} for this file.
• sync - A sync marker to denote end of the header.

Uncompressed SequenceFile Format

• Header
• Record
  • Record length
  • Key length
  • Key
  • Value
• A sync-marker every few 100 bytes or so.

Record-Compressed SequenceFile Format

• Header
• Record
  • Record length
  • Key length
  • Key
  • Compressed Value
• A sync-marker every few 100 bytes or so.

Block-Compressed SequenceFile Format

• Header
• Record Block
  • Compressed key-lengths block-size
  • Compressed key-lengths block
  • Compressed keys block-size
  • Compressed keys block
  • Compressed value-lengths block-size
  • Compressed value-lengths block
  • Compressed values block-size
  • Compressed values block
• A sync-marker every few 100 bytes or so.

The compressed blocks of key lengths and value lengths consist of the actual lengths of individual keys/values encoded in ZeroCompressedInteger format.