VM guest or applications report inconsistencies or I/O errors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Migrating the VM fails with error : "Error caused by file /vmfs/volumes/5c48242b-#######-####-20677ce7ecf8/#################_replica/##################.vmdk

• One or more VMs fail to boot, or on boot he OS report filesystem inconsistencies or corruption.

• At guest/application level, I/O errors or data inconsistency may be encountered within VMs.  

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The storage disk has failed at the hardware level, causing the LUN to report medium errors due to its inability to properly read or write data.

Reference:

From ESXi, /var/log/vmkernel.log  reports medium errors against the LUN on which the VM resides.

Wa() vmkwarning: cpu4:2097808)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x28 (0x45b97faee3c0, 2098738) to dev "naa.####################" on path "vmhba1:C1:T0:L2" Failed: Wa() vmkwarning: cpu4:<cpu-id>)WARNING: HPP: HppScsiThrottleLogForDevice

Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x0 0x0. hppAction = 1 In() vmkernel: cpu4:2097808)ScsiDeviceIO: 4686: Cmd(0x45b97faee3c0) 0x28, CmdSN 0x7324f41 from world <world-ID> to dev "naa.########################" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x0 0x0 Medium Error, LBA

Resolution

A medium error (SCSI code: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x0 0x0) indicates an error on the physical medium (device, drive) at the storage array level, which is reported by the array. 

For such medium errors, engage your storage array vendor to investigate further.