SYNOPSIS

 To start:
      bin/start-balancer.sh [-threshold ]
      Example: bin/ start-balancer.sh 
                     start the balancer with a default threshold of 10%
               bin/ start-balancer.sh -threshold 5
                     start the balancer with a threshold of 5%
 To stop:
      bin/ stop-balancer.sh
 

DESCRIPTION

The threshold parameter is a fraction in the range of (0%, 100%) with a default value of 10%. The threshold sets a target for whether the cluster is balanced. A cluster is balanced if for each datanode, the utilization of the node (ratio of used space at the node to total capacity of the node) differs from the utilization of the (ratio of used space in the cluster to total capacity of the cluster) by no more than the threshold value. The smaller the threshold, the more balanced a cluster will become. It takes more time to run the balancer for small threshold values. Also for a very small threshold the cluster may not be able to reach the balanced state when applications write and delete files concurrently.

The tool moves blocks from highly utilized datanodes to poorly utilized datanodes iteratively. In each iteration a datanode moves or receives no more than the lesser of 10G bytes or the threshold fraction of its capacity. Each iteration runs no more than 20 minutes. At the end of each iteration, the balancer obtains updated datanodes information from the namenode.

A system property that limits the balancer's use of bandwidth is defined in the default configuration file:

 
   dfs.balance.bandwidthPerSec
   1048576
   Specifies the maximum bandwidth that each datanode 
 can utilize for the balancing purpose in term of the number of bytes 
 per second. 
 
 

This property determines the maximum speed at which a block will be moved from one datanode to another. The default value is 1MB/s. The higher the bandwidth, the faster a cluster can reach the balanced state, but with greater competition with application processes. If an administrator changes the value of this property in the configuration file, the change is observed when HDFS is next restarted.

MONITERING BALANCER PROGRESS

After the balancer is started, an output file name where the balancer progress will be recorded is printed on the screen. The administrator can monitor the running of the balancer by reading the output file. The output shows the balancer's status iteration by iteration. In each iteration it prints the starting time, the iteration number, the total number of bytes that have been moved in the previous iterations, the total number of bytes that are left to move in order for the cluster to be balanced, and the number of bytes that are being moved in this iteration. Normally "Bytes Already Moved" is increasing while "Bytes Left To Move" is decreasing.

Running multiple instances of the balancer in an HDFS cluster is prohibited by the tool.

The balancer automatically exits when any of the following five conditions is satisfied:

1. The cluster is balanced;
2. No block can be moved;
3. No block has been moved for five consecutive iterations;
4. An IOException occurs while communicating with the namenode;
5. Another balancer is running.

Upon exit, a balancer returns an exit code and prints one of the following messages to the output file in corresponding to the above exit reasons:

1. The cluster is balanced. Exiting
2. No block can be moved. Exiting...
3. No block has been moved for 3 iterations. Exiting...
4. Received an IO exception: failure reason. Exiting...
5. Another balancer is running. Exiting...

The administrator can interrupt the execution of the balancer at any time by running the command "stop-balancer.sh" on the machine where the balancer is running.]]> true if exclusive locks are supported or false otherwise. @throws IOException @see StorageDirectory#lock()]]> Local storage information is stored in a separate file VERSION. It contains type of the node, the storage layout version, the namespace id, and the fs state creation time.

Local storage can reside in multiple directories. Each directory should contain the same VERSION file as the others. During startup Hadoop servers (name-node and data-nodes) read their local storage information from them.

The servers hold a lock for each storage directory while they run so that other nodes were not able to startup sharing the same storage. The locks are released when the servers stop (normally or abnormally).]]> Removes contents of the current directory and creates an empty directory. This does not fully format storage directory. It cannot write the version file since it should be written last after all other storage type dependent files are written. Derived storage is responsible for setting specific storage values and writing the version file to disk. @throws IOException]]> Locking is not supported by all file systems. E.g., NFS does not consistently support exclusive locks.

If locking is supported we guarantee exculsive access to the storage directory. Otherwise, no guarantee is given. @throws IOException if locking fails]]> stream of bytes (of BLOCK_SIZE or less) This info is stored on a local disk. The DataNode reports the table's contents to the NameNode upon startup and every so often afterwards. DataNodes spend their lives in an endless loop of asking the NameNode for something to do. A NameNode cannot connect to a DataNode directly; a NameNode simply returns values from functions invoked by a DataNode. DataNodes maintain an open server socket so that client code or other DataNodes can read/write data. The host/port for this server is reported to the NameNode, which then sends that information to clients or other DataNodes that might be interested.]]> @see Storage]]> " Many of the activity metrics are sampled and averaged on an interval which can be specified in the metrics config file.

For the metrics that are sampled and averaged, one must specify a metrics context that does periodic update calls. Most metrics contexts do. The default Null metrics context however does NOT. So if you aren't using any other metrics context then you can turn on the viewing and averaging of sampled metrics by specifying the following two lines in the hadoop-meterics.properties file:

        dfs.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
        dfs.period=10
  

Note that the metrics are collected regardless of the context used. The context with the update thread is used to average the data periodically Impl details: We use a dynamic mbean that gets the list of the metrics from the metrics registry passed as an argument to the constructor]]> This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #blocksRead}.inc()]]> Data Node runtime statistic info is report in another MBean @see org.apache.hadoop.hdfs.server.datanode.metrics.DataNodeStatisticsMBean]]> TreeSet]]> :/data[/] HTTP/1.1 }]]> =0.16)]]> The purpose of registration is to identify whether the new datanode serves a new data storage, and will report new data block copies, which the namenode was not aware of; or the datanode is a replacement node for the data storage that was previously served by a different or the same (in terms of host:port) datanode. The data storages are distinguished by their storageIDs. When a new data storage is reported the namenode issues a new unique storageID.

Finally, the namenode returns its namespaceID as the registrationID for the datanodes. namespaceID is a persistent attribute of the name space. The registrationID is checked every time the datanode is communicating with the namenode. Datanodes with inappropriate registrationID are rejected. If the namenode stops, and then restarts it can restore its namespaceID and will continue serving the datanodes that has previously registered with the namenode without restarting the whole cluster. @see org.apache.hadoop.hdfs.server.datanode.DataNode#register()]]> blocklist) and (block-->machinelist) tables.]]> no further work needed here. 2. Removed from hosts --> mark AdminState as decommissioned. 3. Added to exclude --> start decommission. 4. Removed from exclude --> stop decommission.]]> blocklist (kept on disk, logged) 2) Set of all valid blocks (inverted #1) 3) block --> machinelist (kept in memory, rebuilt dynamically from reports) 4) machine --> blocklist (inverted #2) 5) LRU cache of updated-heartbeat machines]]> :/listPaths[/][[&option]*] HTTP/1.1 } Where option (default) in: recursive ("no") filter (".*") exclude ("\..*\.crc") Response: A flat list of files/directories in the following format: {@code

{@link #filesTotal}.set()]]> For the metrics that are sampled and averaged, one must specify a metrics context that does periodic update calls. Most metrics contexts do. The default Null metrics context however does NOT. So if you aren't using any other metrics context then you can turn on the viewing and averaging of sampled metrics by specifying the following two lines in the hadoop-meterics.properties file:

        dfs.class=org.apache.hadoop.metrics.spi.NullContextWithUpdateThread
        dfs.period=10
  

Note that the metrics are collected regardless of the context used. The context with the update thread is used to average the data periodically Impl details: We use a dynamic mbean that gets the list of the metrics from the metrics registry passed as an argument to the constructor]]> This class has a number of metrics variables that are publicly accessible; these variables (objects) have methods to update their values; for example:

{@link #syncs}.inc()]]> datanode whose total size is equal to size @param datanode a data node @param size requested size @return a list of blocks & their locations @throws RemoteException if size is less than or equal to 0 or datanode does not exist]]> The tool scans all files and directories, starting from an indicated root path. The following abnormal conditions are detected and handled:

• files with blocks that are completely missing from all datanodes.
  In this case the tool can perform one of the following actions:
  • none ({@link org.apache.hadoop.hdfs.server.namenode.NamenodeFsck#FIXING_NONE})
  • move corrupted files to /lost+found directory on DFS ({@link org.apache.hadoop.hdfs.server.namenode.NamenodeFsck#FIXING_MOVE}). Remaining data blocks are saved as a block chains, representing longest consecutive series of valid blocks.
  • delete corrupted files ({@link org.apache.hadoop.hdfs.server.namenode.NamenodeFsck#FIXING_DELETE})
• detect files with under-replicated or over-replicated blocks