Windows VM performance degradation after storage controller failover due to CRC errors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• Windows VMs may report instability due to performance degradation .
• Application-level outages or high latency reported by users across multiple VMs.
• Storage array or switch metrics indicate incrementing Cyclic Redundancy Check (CRC) errors on specific physical ports.
• The ESXi vmkernel.log contains frequent "deteriorated performance" warnings:

 /var/run/log/vmkernel.log 

ScsiDeviceIO: 1781: Device naa.#### performance has deteriorated. I/O latency increased from average value of #### microseconds to #### microseconds.

 

Environment

VMware vSphere ESXi 8.x

Cause

• The cause is physical Layer 1 signal corruption on the fiber optic cabling or SFPs. In multipath environments, these errors may remain hidden if the degraded path is secondary or standby. When a controller failover occurs, the ESXi host is forced to use the faulty link, resulting in frame corruption, retransmissions, and high SCSI I/O latency.
• Dropped frame events are reported uniquely across various common generic and vendor-specific drivers, as demonstrated in the command output below:

$ localcli storage san fc events get

FC Event Log 

####-##-## ##:##:##.### [vmhba2] LINK UP
####-##-## ##:##:##.### [vmhba3] Dropped frames (67211 of 402 bytes) on C0:T0:L2 cmd:0x28
####-##-## ##:##:##.### [vmhba3] Dropped frames (102496 of 402 bytes)on C0:T1:L2 cmd:0x28
####-##-## ##:##:##.### [vmhba3] Dropped frames (82026 of 402 bytes) on C0:T0:L2 cmd:0x28
####-##-## ##:##:##.### [vmhba3] Dropped frames (92416 of 402 bytes) on C0:T0:L2 cmd:0x28

Resolution

1. Identify the Faulty Path: Review ESXi host HBA statistics for incrementing CRC errors using the following command: esxcli network nic stats get -n vmnicX (Note: Replace vmnicX with the relevant uplink identifier)
2. Verify Switch Statistics: Inspect the physical Fibre Channel switch ports connected to the host and storage for Invalid CRC Count or Receive CRC errors.
3. Replace Physical Components: Replace the fiber optic patch cables and SFP modules for the identified faulty ports in the affected datacenters.
4. Baseline Monitoring: Clear the interface statistics on the switch and the ESXi host. Monitor the environment for 24 hours to ensure CRC counts remain at zero and latency returns to the expected baseline (typically 3–5ms).