Guests on vSAN experience severe latency

Products

• VMware vSAN

Issue/Introduction

Guest VMs residing on vSAN experience severe storage latency

Environment

ESXi with vSAN OSA storage

Cause

ESXi is experiencing severe I/O latency to vSAN disks - vmkernel.log contains entries similar to the following:

YYYY-MM-DDTHH:MM:SS.653Z In(182) vmkernel: cpu33:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 5671 msec status=Success
YYYY-MM-DDTHH:MM:SS.582Z In(182) vmkernel: cpu33:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 14950 msec status=Success
YYYY-MM-DDTHH:MM:SS.684Z In(182) vmkernel: cpu33:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 7193 msec status=Success
YYYY-MM-DDTHH:MM:SS.591Z In(182) vmkernel: cpu33:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 6080 msec status=Success
YYYY-MM-DDTHH:MM:SS.162Z In(182) vmkernel: cpu38:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 4021 msec status=Success
YYYY-MM-DDTHH:MM:SS.002Z In(182) vmkernel: cpu38:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 4339 msec status=Success
YYYY-MM-DDTHH:MM:SS.322Z In(182) vmkernel: cpu32:2101098 opID=901e0973)FS3Misc: 3208: Slow IO latency FS3_DeviceIO 9738 msec status=Success

Resolution

Verify that storage controller is on the vSAN Virtual Compatibility Guide and on the correct driver & firmware pairing for the version of ESXi in use and update accordingly.  If the problem remains, engage the hardware vendor to run hardware diagnostics on the server, storage controller and disks