big-data/hadoop
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HDFS-17345. Add a metrics to record block report generating cost time. (#6475). Contributed by farmmamba.

Browse Source 
Reviewed-by:  Shuyan Zhang <zhangshuyan@apache.org>
Signed-off-by:  Shuyan Zhang <zhangshuyan@apache.org>
This commit is contained in:
hfutatzhanghb 2024-03-06 16:59:00 +08:00 committed by GitHub
parent 5584efd8d4
commit 7012986fc3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 8 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
View File

@ -458,6 +458,8 @@ Each metrics record contains tags such as SessionId and Hostname as additional i
| `BlockReportsAvgTime` | Average time of block report operations in milliseconds | | `BlockReportsAvgTime` | Average time of block report operations in milliseconds |
| `BlockReports`*ServiceId*`-`*NNId*`NumOps` | Total number of block report operations to specific serviceId and nnId | | `BlockReports`*ServiceId*`-`*NNId*`NumOps` | Total number of block report operations to specific serviceId and nnId |
| `BlockReports`*ServiceId*`-`*NNId*`AvgTime` | Average time of block report operations to specific serviceId and nnId in milliseconds | | `BlockReports`*ServiceId*`-`*NNId*`AvgTime` | Average time of block report operations to specific serviceId and nnId in milliseconds |
| `BlockReportsCreateCostMillsNumOps` | Total number of block report creating operations |
| `BlockReportsCreateCostMillsAvgTime` | Average time of block report creating operations in milliseconds |
| `IncrementalBlockReportsNumOps` | Total number of incremental block report operations | | `IncrementalBlockReportsNumOps` | Total number of incremental block report operations |
| `IncrementalBlockReportsAvgTime` | Average time of incremental block report operations in milliseconds | | `IncrementalBlockReportsAvgTime` | Average time of incremental block report operations in milliseconds |
| `IncrementalBlockReports`*ServiceId*`-`*NNId*`NumOps` | Total number of incremental block report operations to specific serviceId and nnId | | `IncrementalBlockReports`*ServiceId*`-`*NNId*`NumOps` | Total number of incremental block report operations to specific serviceId and nnId |

1
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
View File

@ -452,6 +452,7 @@ List<DatanodeCommand> blockReport(long fullBrLeaseId) throws IOException {
// Log the block report processing stats from Datanode perspective // Log the block report processing stats from Datanode perspective
long brSendCost = monotonicNow() - brSendStartTime; long brSendCost = monotonicNow() - brSendStartTime;
long brCreateCost = brSendStartTime - brCreateStartTime; long brCreateCost = brSendStartTime - brCreateStartTime;
dn.getMetrics().addBlockReportCreateCost(brCreateCost);
dn.getMetrics().addBlockReport(brSendCost, getRpcMetricSuffix()); dn.getMetrics().addBlockReport(brSendCost, getRpcMetricSuffix());
final int nCmds = cmds.size(); final int nCmds = cmds.size();
LOG.info((success ? "S" : "Uns") + LOG.info((success ? "S" : "Uns") +

5
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
View File

@ -132,6 +132,7 @@ public class DataNodeMetrics {
@Metric MutableRate heartbeatsTotal; @Metric MutableRate heartbeatsTotal;
@Metric MutableRate lifelines; @Metric MutableRate lifelines;
@Metric MutableRate blockReports; @Metric MutableRate blockReports;
@Metric private MutableRate blockReportsCreateCostMills;
@Metric MutableRate incrementalBlockReports; @Metric MutableRate incrementalBlockReports;
@Metric MutableRate cacheReports; @Metric MutableRate cacheReports;
@Metric MutableRate packetAckRoundTripTimeNanos; @Metric MutableRate packetAckRoundTripTimeNanos;
@ -321,6 +322,10 @@ public void addBlockReport(long latency, String rpcMetricSuffix) {
} }
} }
public void addBlockReportCreateCost(long latency) {
blockReportsCreateCostMills.add(latency);
}
public void addIncrementalBlockReport(long latency, public void addIncrementalBlockReport(long latency,
String rpcMetricSuffix) { String rpcMetricSuffix) {
incrementalBlockReports.add(latency); incrementalBlockReports.add(latency);