hadoop.hdfs.configuration.version 1 version of this configuration file dfs.namenode.logging.level info The logging level for dfs namenode. Other values are "dir"(trac e namespace mutations), "block"(trace block under/over replications and block creations/deletions), or "all". dfs.namenode.secondary.http-address 0.0.0.0:50090 The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.address 0.0.0.0:50010 The address where the datanode server will listen to. If the port is 0 then the server will start on a free port. dfs.datanode.http.address 0.0.0.0:50075 The datanode http server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.ipc.address 0.0.0.0:50020 The datanode ipc server address and port. If the port is 0 then the server will start on a free port. dfs.datanode.handler.count 3 The number of server threads for the datanode. dfs.namenode.http-address 0.0.0.0:50070 The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. dfs.https.enable false Decide if HTTPS(SSL) is supported on HDFS dfs.client.https.need-auth false Whether SSL client certificate authentication is required dfs.https.server.keystore.resource ssl-server.xml Resource file from which ssl server keystore information will be extracted dfs.client.https.keystore.resource ssl-client.xml Resource file from which ssl client keystore information will be extracted dfs.datanode.https.address 0.0.0.0:50475 dfs.namenode.https-address 0.0.0.0:50470 dfs.datanode.dns.interface default The name of the Network Interface from which a data node should report its IP address. dfs.datanode.dns.nameserver default The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes. dfs.namenode.backup.address 0.0.0.0:50100 The backup node server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.backup.http-address 0.0.0.0:50105 The backup node http server address and port. If the port is 0 then the server will start on a free port. dfs.namenode.replication.considerLoad true Decide if chooseTarget considers the target's load or not dfs.default.chunk.view.size 32768 The number of bytes to view for a file on the browser. dfs.datanode.du.reserved 0 Reserved space in bytes per volume. Always leave this much space free for non dfs use. dfs.namenode.name.dir file://${hadoop.tmp.dir}/dfs/name Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. dfs.namenode.fs-limits.max-component-length 0 Defines the maximum number of characters in each component of a path. A value of 0 will disable the check. dfs.namenode.fs-limits.max-directory-items 0 Defines the maximum number of items that a directory may contain. A value of 0 will disable the check. dfs.namenode.edits.dir ${dfs.namenode.name.dir} Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.name.dir dfs.web.ugi webuser,webgroup The user account used by the web interface. Syntax: USERNAME,GROUP1,GROUP2, ... dfs.permissions.enabled true If "true", enable permission checking in HDFS. If "false", permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. dfs.permissions.superusergroup supergroup The name of the group of super-users. dfs.block.access.token.enable false If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanodes. dfs.block.access.key.update.interval 600 Interval in minutes at which namenode updates its access keys. dfs.block.access.token.lifetime 600 The lifetime of access tokens in minutes. dfs.datanode.data.dir file://${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. dfs.datanode.data.dir.perm 700 Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. dfs.replication 3 Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. dfs.replication.max 512 Maximal block replication. dfs.namenode.replication.min 1 Minimal block replication. dfs.blocksize 67108864 The default block size for new files. dfs.client.block.write.retries 3 The number of retries for writing blocks to the data nodes, before we signal failure to the application. dfs.client.block.write.replace-datanode-on-failure.enable ture If there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wise property to enable/disable the feature. See also dfs.client.block.write.replace-datanode-on-failure.policy dfs.client.block.write.replace-datanode-on-failure.policy DEFAULT This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended. dfs.blockreport.intervalMsec 21600000 Determines block reporting interval in milliseconds. dfs.blockreport.initialDelay 0 Delay for first block report in seconds. dfs.datanode.directoryscan.interval 21600 Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk. dfs.datanode.directoryscan.threads 1 How many threads should the threadpool used to compile reports for volumes in parallel have. dfs.datanode.block.volume.choice.policy org.apache.hadoop.hdfs.server.datanode.RoundRobinVolumesPolicy The policy class to use to determine into which of the datanode's available volumes a block must be written to. Default is a simple round-robin policy that chooses volumes in a cyclic order. dfs.heartbeat.interval 3 Determines datanode heartbeat interval in seconds. dfs.namenode.handler.count 10 The number of server threads for the namenode. dfs.namenode.safemode.threshold-pct 0.999f Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. dfs.namenode.safemode.min.datanodes 0 Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. dfs.namenode.safemode.extension 30000 Determines extension of safe mode in milliseconds after the threshold level is reached. dfs.datanode.balance.bandwidthPerSec 1048576 Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. dfs.hosts Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. dfs.hosts.exclude Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. dfs.namenode.max.objects 0 The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. dfs.namenode.decommission.interval 30 Namenode periodicity in seconds to check if decommission is complete. dfs.namenode.decommission.nodes.per.interval 5 The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval. dfs.namenode.replication.interval 3 The periodicity in seconds with which the namenode computes repliaction work for datanodes. dfs.namenode.accesstime.precision 3600000 The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. dfs.datanode.plugins Comma-separated list of datanode plug-ins to be activated. dfs.namenode.plugins Comma-separated list of namenode plug-ins to be activated. dfs.stream-buffer-size 4096 The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. dfs.bytes-per-checksum 512 The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size dfs.client-write-packet-size 65536 Packet size for clients to write dfs.namenode.checkpoint.dir file://${hadoop.tmp.dir}/dfs/namesecondary Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy. dfs.namenode.checkpoint.edits.dir ${dfs.namenode.checkpoint.dir} Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directoires then teh edits is replicated in all of the directoires for redundancy. Default value is same as dfs.namenode.checkpoint.dir dfs.namenode.checkpoint.period 3600 The number of seconds between two periodic checkpoints. dfs.namenode.checkpoint.txns 40000 The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every 'dfs.namenode.checkpoint.txns' transactions, regardless of whether 'dfs.namenode.checkpoint.period' has expired. dfs.namenode.checkpoint.check.period 60 The SecondaryNameNode and CheckpointNode will poll the NameNode every 'dfs.namenode.checkpoint.check.period' seconds to query the number of uncheckpointed transactions. dfs.namenode.num.checkpoints.retained 2 The number of image checkpoint files that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained. dfs.namenode.delegation.key.update-interval 86400000 The update interval for master key for delegation tokens in the namenode in milliseconds. dfs.namenode.delegation.token.max-lifetime 604800000 The maximum lifetime in milliseconds for which a delegation token is valid. dfs.namenode.delegation.token.renew-interval 86400000 The renewal interval for delegation token in milliseconds. dfs.datanode.failed.volumes.tolerated 0 The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. dfs.image.compress false Should the dfs image be compressed? dfs.image.compression.codec org.apache.hadoop.io.compress.DefaultCodec If the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs. dfs.image.transfer.bandwidthPerSec 0 Specifies the maximum amount of bandwidth that can be utilized for image transfer in term of the number of bytes per second. A default value of 0 indicates that throttling is disabled. dfs.namenode.support.allow.format true Does HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS. dfs.datanode.max.transfer.threads 4096 Specifies the maximum number of threads to use for transferring data in and out of the DN. dfs.web.authentication.kerberos.principal HTTP/${dfs.web.hostname}@${kerberos.realm} The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos HTTP SPENGO specification. dfs.web.authentication.kerberos.keytab ${user.home}/dfs.web.keytab The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.