All hosts in the vSAN cluster have checksum errors

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

All host in the vSAN cluster have checksum errors. Failed vSAN Disk were replaced. The disk groups were removed and recreated on each host.
vSAN skyline health shows healthy but checksum errors are still reported in the logs.

e.g vobd.log

2025-04-02T06:25:03.546Z: [vSANCorrelator] 606683026821us: [vob.vsan.dom.unrecoverableerror] vSAN detected an unrecoverable medium or checksum error for component 4ea3e667-863d-348f-39b2-############# on disk group 52d3658b-f00b-5760-68f3-###############.

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

Failed vSAN disk and host memory controller scrubbing errors were identified, which  lead to checksum errors on the vSAN components.

vmkernel.log

2025-03-16T00:38:49.875Z cpu15:2571527)MCA: 209: CE Intr G0 Bf S8c0000400008### A134389#### M900004000400### P1343895####/40 Memory Controller Scrubbing Error on Channel 0.

2025-03-17T20:30:16.634Z: [scsiCorrelator] 135480189560us: [esx.problem.scsi.device.state.permanentloss] Device: naa.############### has been removed or is permanently inaccessible. Affected datastores (if any): Unknown. 

Resolution

• Engage Hardware Vendor for further troubleshooting/fix/replace failed hardware.
  
  
• Open a Ticket with VMware by Broadcom Support to help cleanup vSAN objects with the unrecoverable checksum errors.  

Additional Information

VMware vSAN disk encounters medium errors but not failed out by vSAN