ESXi hosts report and VMs experience storage latency during guest OS patching or periods of heavy use.

Products

• VMware vSphere ESXi

Issue/Introduction

Disk latency is observed when VMs are undergoing guest OS patching or other periods of high read/write IO.

Environment

ESXi (all versions)
Fiber Channel Storage

Cause

• Storage array or fabric issues are leading to latency accessing storage.
• vmkernel log indicates latency increase messages across multiple LUNs hundreds of times:

YYYY-MM-DDThh:mm:ss.000Z cpu23:2098485)WARNING: ScsiDeviceIO: 1513: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 539 microseconds to 149556 microseconds.

• vmkernel log indicates HBA resets:

YYYY-MM-DDThh:mm:ss.000Z cpu51:2097293)ScsiDeviceIO: 4087: Cmd(0x#############) 0x88, cmdId.initiator=0x############# CmdSN0xffffc186badb3a00 from world ####### to dev "naa.################################" failed H:0x8 D:0x0 P:0x0 Cancelled from
YYYY-MM-DDThh:mm:ss.000Z cpu51:2097293)device layer

• vmkernel log indicates fabric path discovery for the same paths every 5 minutes during the latency period:

YYYY-MM-DDThh:mm:ss.000Z cpu50:2097623)qlnativefc: vmhba0(13:0.0): qlnativefcDiscover:1853:creating path vmhba0:0:0:0
YYYY-MM-DDThh:mm:ss.000Z cpu48:2097623)qlnativefc: vmhba0(13:0.0): qlnativefcDiscover:1853:creating path vmhba0:0:1:0

Resolution

This is an environmental storage issue. Contact the fabric and array vendors to perform health check and monitor performance while latency is present.

Additional Information

Interpreting SCSI Sense Codes in ESXi