Receiving a SCSI Medium Error When Performing I/O to a Specific LBA on a LUN

Products

VMware vSphere ESXi

Issue/Introduction

When performing I/O operations on a LUN, the ESXi host reports SCSI Medium Errors in /var/log/vmkernel.log. These errors occur when the host attempts to read data from a specific Logical Block Address (LBA) on the storage device.
Example Log Entries
vmkernel.log YYYY-MM-DDTHH:MM:SSZ cpu4:2098170)ScsiDeviceIO: 4176: Cmd(0x45ba4164fc48) 0x28, CmdSN 0x5 from world 2103375 to dev "naa.600508b1001c####################" failed H:0x0 D:0x2 P:0x0 Valid sense data: **0x3 0x11 0x0** Medium Error, LBA: 15475715
YYYY-MM-DDTHH:MM:SSZ cpu1:2097804)ScsiDeviceIO: 4167: Cmd(0x45b8c12f8348) 0x28, CmdSN 0x334 from world 2099409 to dev "naa.614187705578####################" failed H:0x0 D:0x2 P:0x0 Valid sense data: **0x3 0x0 0x0** Medium Error, LBA: 31915520
hostd.log (During Backup/NFC Operations) YYYY-MM-DDTHH:MM:Sec.MsecZ warning hostd[2100761] [Originator@6876 sub=Libs opID=nbdmode-00000062########] [NFC ERROR]NfcAioLogFatalSessionErrorLocked: A fatal session error occurred. The error was: 'NFC_SESSION_ERROR' (8)
YYYY-MM-DDTHH:MM:Sec.MsecZ error hostd[######] [Originator@6876 sub=Nfcsvc opID=nbdmode-00000062########] Read error from the nfcLib: NFC_SESSION_ERROR (done = yep)
Search for Medium Errors and Unrecovered Read errors in the vmkernel log
```
# grep -E "Sense Data|Medium Error|Unrecovered" /var/log/vmkernel.log
```

Environment

VMware vSphere ESXi

Cause

This error is triggered when the storage controller returns a CHECK CONDITION with MEDIUM ERROR sense data. It explicitly indicates that the physical storage media has degraded or encountered unrecoverable bad blocks at the hardware level.

SCSI Sense Data Breakdown

Sense Key	ASC / ASCQ	Description
0x3 (Medium Error)	`0x11 / 0x00`	Unrecovered Read Error: The device is unable to read data from the requested LBA.
0x3 (Medium Error)	`0x00 / 0x00`	Medium Error: General unrecoverable read failure.

Note: ESXi is operating correctly by surfacing this hardware-level failure upstream. The host issued the command, and the hardware device explicitly failed it.

Impact

Data Integrity: Data residing on the affected LBAs is physically unreadable and likely corrupted.
Virtual Machine Operations: Storage vMotion, VM Cloning, and Snapshot Consolidation will fail consistently when they reach the affected LBA.
Backups: Third-party backups will fail with NFC_SESSION_ERROR during sequential disk reads.

Resolution

Since the issue is rooted in physical hardware, VMware cannot bypass the read error. Follow these steps to resolve the issue:

Identify the Backing Device: Use the naa ID from the logs to identify the affected LUN or local disk: esxcli storage core device list -d naa.<ID_FROM_LOGS>
Halt Migrations: Stop any active Storage vMotion or cloning tasks on the affected datastore to prevent further RAID degradation.
Engage Hardware Vendor: Provide the logs and the naa ID to your storage array or server hardware vendor (e.g., Dell, HP, NetApp) to identify and replace the failing physical disks.
Verify Data Integrity: After the hardware is repaired/replaced, perform a file system consistency check (e.g., chkdsk or fsck) within the guest OS of any VMs residing on the affected LBAs.
Restore from Backup: If the data at the failed LBAs is critical and unreadable after hardware repair, you must restore the affected virtual machine or VMDK from a known-good backup predating the hardware failure.