From cede33997f7ab09fc046017508b680e282289ce3 Mon Sep 17 00:00:00 2001 From: Xiao Chen Date: Tue, 14 Aug 2018 11:56:51 -0700 Subject: [PATCH] HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. --- .../hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md index 6ae20862d5..2e8cbbd618 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md @@ -107,10 +107,10 @@ Deployment This means that when reading and writing striped files, most operations are off-rack. Network bisection bandwidth is thus very important. - For rack fault-tolerance, it is also important to have at least as many racks as the configured EC stripe width. - For EC policy RS (6,3), this means minimally 9 racks, and ideally 10 or 11 to handle planned and unplanned outages. - For clusters with fewer racks than the stripe width, HDFS cannot maintain rack fault-tolerance, but will still attempt - to spread a striped file across multiple nodes to preserve node-level fault-tolerance. + For rack fault-tolerance, it is also important to have enough number of racks, so that on average, each rack holds number of blocks no more than the number of EC parity blocks. A formula to calculate this would be (data blocks + parity blocks) / parity blocks, rounding up. + For EC policy RS (6,3), this means minimally 3 racks (calculated by (6 + 3) / 3 = 3), and ideally 9 or more to handle planned and unplanned outages. + For clusters with fewer racks than the number of the parity cells, HDFS cannot maintain rack fault-tolerance, but will still attempt + to spread a striped file across multiple nodes to preserve node-level fault-tolerance. For this reason, it is recommended to setup racks with similar number of DataNodes. ### Configuration keys