Seeing High I/O from Storage side on the Storage DRS Cluster / Frequent Storage Volume Disconnection

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• Using Fibre Channel, seeing High I/O from Storage side on the Storage DRS Cluster. 
• The Latencies are seen within the Applications or in general within the VMs, while nothing has been changed in the environment
• Permanent or frequent Storage Volume disconnection(s) from LUN(s) 

Environment

ESXi 7.x

ESXi 8.x

Cause

The ESXi Host Logs are showing Latencies relation to the devices being accessed. Examples:

 

/var/log/vmkernel.log:

2025-03-20T19:03:05.361Z cpu81:2098508)WARNING: ScsiDeviceIO: 1513: Device ######## performance has deteriorated. I/O latency increased from average value of 1019 microseconds to 95139 microseconds.

 

/var/log/vmkwarning.log:

2025-03-20T19:04:40.833Z cpu49:2098506)WARNING: ScsiDeviceIO: 1513: Device ######## performance has deteriorated. I/O latency increased from average value of 995 microseconds to 43345 microseconds.

 

If the LUNs are underperforming were the VMs reside this results in VMs experiencing latency.

Resolution

Please see KB "performance has deteriorated" messages in ESXi host logs

Engage the storage array vendor for further investigation for the poorly performing devices