Unable to clone, backup, or consolidate VMs error "Checksum mismatch"

Products

VMware vSAN

Issue/Introduction

Clones, backups, or consolidation operations of VMs are failing with the following messaging

From Hostd.log

hostd.log:message = "Failed to copy source (/vmfs/volumes/vsan:/namespace/VM_12.vmdk) to destination (/vmfs/volumes/vsan:/namesapce/VM_12.vmdk): Checksum mismatch. "

2025-04-28T06:00:34.299Z: [vSANCorrelator] 4631684299497us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component f7842761-01b3-f8eb-0a93-########### on disk group 520f9cd0-0f4c-67e6-cbbf-###########.

From vmkernel.log

2023-01-08T05:32:02.042Z cpu7:2099333)WARNING: LSOM: LSOMReadVerifyChecksum:4397: Throttled: Checksum error detected on component 3134ba63-####-####-####-########910, comp offset 210520969216 (computed CRC 0x0 != saved CRC 0x81bf8868 (faked: Y))
2023-01-05T19:03:27.576Z: [vSANCorrelator] 4820779458341us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component 9c0cb35e-####-####-####-########610 on disk group 523a74b9-####-####-####-########f09.
2022-10-23T03:15:13.877Z: [vSANCorrelator] 22532796272us: [vob.vsan.dom.errorfixed] vSAN detected and fixed a medium or checksum error for component 938c5363-####-####-####-########610 on disk group 52106776-####-####-####-########862.

Environment

VMware vSAN (All Versions)

Cause

Medium errors are present on the physical disk where the component with checksum errors resides.

From Hostd.log

2025-04-28T06:00:34.299Z: [vSANCorrelator] 4631684299497us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component f7842761-01b3-f8eb-0a93-########### on disk group 520f9cd0-0f4c-67e6-cbbf-###########.
2025-04-28T15:15:46.534Z: [vSANCorrelator] 4664996091018us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component f7842761-01b3-f8eb-0a93-########### on disk group 520f9cd0-0f4c-67e6-cbbf-###########.
2025-04-28T15:15:46.534Z: [vSANCorrelator] 4665059504699us: [esx.problem.vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component f7842761-01b3-f8eb-0a93-########### on disk group 520f9cd0-0f4c-67e6-cbbf-###########.

From Vmkernel.log

2025-05-05T18:02:42.745Z cpu54:2098074)ScsiDeviceIO: 4167: Cmd(0x45dea00a8308) 0x28, CmdSN 0x6098bd81 from world 0 to dev "naa.############6391" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 2520853716
2025-05-05T18:02:43.284Z cpu30:2098080)ScsiDeviceIO: 4167: Cmd(0x45dea66d48c8) 0x28, CmdSN 0x6098bdbd from world 0 to dev "naa.############6391" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 2520853716
2025-05-05T18:02:43.821Z cpu34:2098082)ScsiDeviceIO: 4167: Cmd(0x45de788f2e88) 0x28, CmdSN 0x6098bdf0 from world 0 to dev "naa.############6391" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 2520853716

From esxcli vsan storage list.

naa.############6391:
Device: naa.############6391
Display Name: naa.############6391
Is SSD: true
VSAN UUID: 52927d10-a1ed-eb4e-d59d-############
VSAN Disk Group UUID: 520f9cd0-0f4c-67e6-cbbf-############

Some different types of medium errors that can potentially be seen in the logs

0x3 0x3 0x0 - PERIPHERAL DEVICE WRITE FAULT

0x3 0x10 0x0 - ID CRC OR ECC ERROR

0x3 0x11 0x0 - Unrecovered read error

0x3 0x31 0x0 - Medium Format corruption

Resolution

If medium errors are encountered in the data region of the disk and the disk wasn't failed out by vSAN you can do one of the following:

Using the preferred method, contact the hardware vendor and get the disk replaced.
Else:
1. If Deduplication is not enabled:
  - Perform a pre-check by navigating to vSAN Cluster > Monitor > Data Migration Pre-check. Under Pre-check Data Migration For, select OBJECT, then go to the problematic host and select the problematic disk (as seen in the logs) that needs to be removed from the disk group and re-added.
    - Remove the failed disk with medium errors from the disk group and then add it back. Upon adding the disk back to the disk group, the bad blocks are automatically reallocated by the disk for non-use, so they don't get used again.
2. If Deduplication is enabled:
  - Remove the disk group which contains the disk failing with medium errors from the host and recreate the disk group. Upon recreating the disk group the bad blocks are automatically reallocated by the disk for non-use, so they don't get used again.
    Note: This results in a resync to rebuild data due to the disk/disk group being removed and then recreated/added back.
3. Backups, clone, consolidation operations may fail when you see these errors "vSAN detected an unrecoverable medium or checksum error for component", check which object is affected and create a new policy with checksum option disabled and apply that policy to Object only and not to the entire VM.

Reference KB

VMware vSAN disk encounters medium errors but not failed out by vSAN