Large I/O block size operations show high latency in Windows

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

In vSphere, when performing a task that utilizes large I/O block sizes in Windows, performance charts in vCenter or ESXTOP display very high device latency or DAVG, even though the actual throughput is excellent.

Cause

Windows 2008 and higher can issue much larger I/O block sizes for certain operations, such as file copy. However, this can be reproduced with other applications, such as IOmeter with large I/O block size test parameters.

This issue can also be reproduced for illustrative purposes in a Linux virtual machine with the dd command and issuing commands with a large block size (i.e. - 1M).

Resolution

High latency is due to a large I/O block size that needs to be split up into smaller I/O sizes in order to be transmitted to the storage device. When this I/O split occurs, the measure of latency is against the entire command, not the individual chunks.

 

Therefore, you must wait for all chunks to complete to the array before reporting back on the latency. This results in a false positive when high latency is observed to the array, when a performance problem does not actually exist.

Note: Reducing the Disk.DiskMaxIOSize advanced setting in the ESX/ESXi host does not improve the latency results because the Guest operating system is the one issuing large I/O block size. For more information, see Tuning ESX/ESXi for better storage performance by modifying the maximum I/O block size.

The Windows registry may be altered to issue smaller I/O block sizes, resulting in lower latency. However, this is merely a false positive. For more information, see:

• Large I/Os Are Split into 64KB Units When Using LSILogic Driver (9645697)
• The Guest Operating System Storage Considerations section in the Performance Best Practices for VMware vSphere 8.0.

While one may think that issuing a virtual machine migration or virtual machine deployment can be considered a large block operation, in reality the vmkernel issues I/Os block size of 32k. So, this is significantly smaller than what the guest operating system issues, resulting in no high latency for this operation.

 

Additional Information

Tuning ESX/ESXi for better storage performance by modifying the maximum I/O block size