VMs with Multiple VMDKs Report High Disk Read/Write Latency - VMware vSphere
Article ID: 392848
Updated On:
Products
Issue/Introduction
Administrators may observe high read and/or write latency reported within a Virtual Machine (VM) (e.g., Windows Guest OS reporting high Avg. Disk Read/Write Latency), while the latency measured at the storage array remains healthy. This symptom is frequently observed in high-I/O virtual machines, such as those running SQL Server, when configured with only a single virtual SCSI controller.
Symptoms can include:
- High disk latency reported inside the Guest OS Performance Monitor or equivalent tools.
- No corresponding latency or performance warnings reported at the Storage Array level.
- SQL Server Error 833: "I/O requests taking longer than 15 seconds to complete."
- VM is configured with only a single SCSI controller handling all virtual disks.
Environment
- VMware vSphere ESXi 6.7, 7.0, and 8.0.
- Guest OS: All supported Windows and Linux distributions.
- Applications: I/O-intensive workloads (e.g., SQL Server, Database instances).
Cause
A single virtual SCSI controller has a finite queue depth. When multiple high-I/O disks (such as OS, Data, Logs, and TempDB) are attached to a single controller, I/O requests queue up within the Guest OS. This artificial bottleneck results in reported latency inside the VM, even when the underlying physical storage array is performing optimally.
Resolution
- Prepare for Maintenance:
- This procedure requires virtual machine downtime to modify the hardware configuration.
- This procedure will not remove any data, but it will alter the virtual disk mapping.
- Please consult your Guest OS vendor regarding any configuration updates required within the guest operating system due to the modified disk mapping
- Ensure that there are no active snapshots on the VM.
- Add Additional PVSCSI Controllers:
- Right-click the Virtual Machine and select Edit Settings.
- Click Add New Device and select SCSI Controller.
- Expand the new controller and ensure the type is set to VMware Paravirtual (PVSCSI) for optimal performance.
- Repeat this step until you have up to four (4) PVSCSI controllers total.
- Distribute Virtual Disks:
- Gracefully power off the guest OS and VM
- In the Edit Settings dialog, locate existing virtual hard disks.
- Change the Virtual Device Node for each disk to map it to one of the newly created PVSCSI controllers.
- Best Practice Strategy is to balance load across all 4 PVSCSI controllers:
- Controller 0: OS Disk.
- Controller 1: SQL Data files.
- Controller 2: SQL Transaction Logs.
- Controller 3: TempDB or secondary workloads.
- Save and Power On:
- Click OK to save the changes.
- Power on the virtual machine and verify that all disks are correctly detected and mounted within the Guest OS.
- Address any configuration recommendations provided by your Guest OS vendor concerning the updated virtual disk mapping.
Additional Information
For further details on maximizing I/O performance, refer to the SQL Server on VMware Best Practices Guide.
Feedback
