vSphere storage device latency performance metrics

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

This article provides information on storage device latency performance metrics.

Resolution

The vSphere performance metrics made available by esxtop, resxtop and the vSphere Client are averages over the sample timeframe. Measurements snapshots are taken at the end of each sample timeframe and values are averaged. The resulting value reported as the average value within that range.

For more information refer to Performance Monitoring Utilities: resxtop and esxtop.

If the sampled metric consists of a non-homogeneous set of data, metrics may be skewed in unexpected ways.

For example:

Measurements of storage device command latency (DAVG/cmd) may generate higher or lower values than expected, if not all commands within the sample complete at the same speed. Most storage device commands complete quickly (< 25ms), but a small number of commands (such as XCOPY, used in VAAI) may take significantly longer to complete.

 

However, If a particular sample window consists of a low number of fast commands and one very slow command, the average among these are higher than the median.

The measured average of storage latency over a sample window varies significantly depending on how many other commands are completing at the same time, even though these examples depict only a single command which is taking a long time to complete:

• 4 commands completed in 1ms and 1 command completed in 500ms implies an average latency of 100.8ms.
• 9 commands completed in 1ms and 1 command completed in 500ms implies an average latency of 50.9ms.
• 19 commands completed in 1ms and 1 command completed in 500ms implies an average latency of 25.95ms.

Note: Consider what is being measured when evaluating averages of any sampled performance metric before drawing any conclusions based on those metrics.