Error: "Data error (cyclic redundancy check)" while taking a backup of the VM.

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Virtual machine backup task failed with errors 'Error (3) Data error (cyclic redundancy check).'

Task failed. Error: Cannot find layout information for disk: 2000

The below error may show up on Veeam backup interface.

The vendor analysis shared as below.

Backup Vendor Log Snippets:
[18.01.2025 22:14:15.732] <47> Error (3) Data error (cyclic redundancy check).
[18.01.2025 22:14:15.732] <47> Error (3) Asynchronous request operation has failed. [requestsize = 4194304] [offset = 68160585728]
[18.01.2025 22:14:15.732] <47> Error (3) Failed to upload disk '>'
[18.01.2025 22:14:15.732] <47> Error (3) Agent failed to process method {DataTransfer.SyncDisk}.
[18.01.2025 22:14:15.732] <47> Error (3) (System.AggregateException)
[18.01.2025 22:14:15.732] <47> Error (3) Data error (cyclic redundancy check). (Veeam.Backup.Common.CCppComponentException)
[18.01.2025 22:14:15.732] <47> Error (3) in c++: Error code: 0x00000017
[18.01.2025 22:14:15.732] <47> Error (3) in c++: Next asynchronous read request cannot be processed.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

The error "Data error (cyclic redundancy check)" reported by backup application.
The affected VM is residing on a datastore and the backend physical disks associated the datastore are experiencing hardware (disks) Medium Error for the hard drives .
When we have underlying disks read errors, it is expected to see the data errors while performing READ/WRITE on disks.
The data/medium errors can be validated by verifying the logs snippets on ESXi host.

/var/log/vmkernel.log

2025-01-18T13:38:24.068Z cpu4:2097457)smartpqi01: pqisrc_show_sense_data_simple:0164: [ERR INFO] BTL: 1:0:2 op=0x28 path=Raid K:C:Q: 3:11:0

2025-01-18T13:38:24.068Z cpu4:2097457)smartpqi01: os_raid_response_error:1219: err_info status=0x2 data_out=0x0. VMK host=0x0 device=0x2

2025-01-18T13:38:24.068Z cpu6:2097722)WARNING: HPP: HppThrottleLogForDevice:1136: Cmd 0x28 (0x45b907ccb408, 2111676) to dev "naa.600508b1001c13f0c5ca8cc9a955362b" on path "vmhba1:C1:T0:L2" Failed:

2025-01-18T13:38:24.068Z cpu6:2097722)WARNING: HPP: HppThrottleLogForDevice:1144: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. hppAction = 1

2025-01-18T13:38:24.068Z cpu6:2097722)ScsiDeviceIO: 4176: Cmd(0x45b907ccb408) 0x28, CmdSN 0x80000012 from world 2111676 to dev "naa.600508b1001c13f0c5ca8cc9a955362b" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA:

/var/log/vmkwarning.log:

2025-01-19T13:36:02.497Z cpu8:2097724)WARNING: HPP: HppThrottleLogForDevice:1136: Cmd 0x28 (0x45b90bb3d8c8, 2111676) to dev "naa.600508b1001c13f0c5ca8cc9a955362b" on path "vmhba1:C1:T0:L2" Failed:

2025-01-19T13:36:02.497Z cpu8:2097724)WARNING: HPP: HppThrottleLogForDevice:1144: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. hppAction = 1

Resolution

Since the Medium errors on the drive is hardware related, please engage hardware vendor to assist with the issue further.

Additional Information

Type	Code	Name	Description
Host Status	[0x0]	OK	This status is returned when there is no error on the host side. This is when you will see if there is a status for a Device or Plugin. It is also when you will see Valid sense data instead of Possible sense Data.
Device Status	[0x2]	CHECK_CONDITION	This status is returned when a command fails for a specific reason. When a CHECK CONDITION is received, the ESX storage stack will send out a SCSI command 0x3 (REQUEST SENSE) in order to get the SCSI sense data (Sense Key, Additional Sense Code, ASC Qualifier, and other bits). The sense data is listed after Valid sense data in the order of Sense Key, Additional Sense Code, and ASC Qualifier.
Plugin Status	[0x0]	GOOD	No error. (ESXi 5.x / 6.x only)
Sense Key	[0x3]	MEDIUM ERROR
Additional Sense Data	11/00	UNRECOVERED READ ERROR