Guest VM OS report high I/O wait times from guest or 3rd party performance monitoring tools

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x
• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

• The Guest OS, regardless of the specific OS, is reporting high I/O wait times from guest performance monitoring tools such as top (Linux) or perfmon (Windows)

• You see messages in the ESXi host logs similar to the following: 

• •  I/O latency on the SAN LUNs

[ from /var/run/log/vmkwarning.log ] 

2025-05-12T20:52:42.242Z Wa(180) vmkwarning: cpu111:2098313)WARNING: ScsiDeviceIO: 1772: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 331 microseconds to 7318 microseconds.

• • Failing SCSI commands with sense data: 0x2 0x4 0x3 (LUN not ready):

[ from /var/run/log/vmkernel.log ]

2025-05-13T19:24:45.797Z In(182) vmkernel: cpu58:2098312)ScsiDeviceIO: 4633: Cmd(0x45b9da7555c0) 0x1a, CmdSN 0x1f6667d from world 0 to dev "naa.################################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3

• • Storage paths degraded messages
    
    [ from /var/run/log/vobd.log ]

2025-09-03T23:29:36.954Z: [scsiCorrelator] 5662461916856us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.############################ degraded. Path vmhba3:C0:T1:L36 is down. Affected datastores: Unknown.

• The LUNs reporting the 0x2 0x4 0x3 sense data may not be the ones where the problem VM is registered. 

Environment

• vSphere ESXi (all versions)

Cause

• The presence of these errors can contribute to overall storage responsiveness, as the commands to the problem LUNs are failing and have to be requeued (and will continue to fail repeatedly until the issue is resolved). The severity of this issue will increase as the number of LUNs having issues increases. This is a SAN-side issue.

Resolution

• Please work with your SAN team and/or SAN vendor to troubleshoot the issues with the LUNs reporting the above sense data.
• If needed, please open a case with Broadcom for further assistance, as well.

 

Additional Information

• The "D:0x2" in the above line indicates that the SAN has reported a "check condition" from the device(target) end and that action should be taken from the SAN end to troubleshoot the problem further. 
• The Sense Data: 0x2 0x4 0x3 indicates a "LUN not ready" error. 
• For more information on Device SCSI Sense Codes in an vSphere environment, see KB: Understanding SCSI Target NMP Errors and Conditions in ESXi
• For official documentation on SCSI commands, codes, responses, etc, see: SCSI Storage Interfaces (INCITS Home of SAS, Technical Committee T10)