Backups with Veeam tool does not work, getting "Error: Data error (cyclic redundancy check)"

Products

• VMware vSAN

Issue/Introduction

Backups are not completing successfully. 

Only certain VMs are impacted.

Veeam backup software failing with below error

"Error: Data error (cyclic redundancy check)".

Asynchronous request operation has failed. [requestsize = 380928] [offset = 36042321920]
Failed to upload disk '>'
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Data error (cyclic redundancy check).
Asynchronous request operation has failed. [requestsize = 380928] [offset = 36042321920]
Unable to retrieve next block transmission command. Number of already processed blocks: [32388].
Failed to download disk

Environment

VMware vSAN (All Versions)

Cause

Storage device has extremely high latency.  This is presented as, "performance has deteriorated" messaging in the vmkernel.log.

2025-04-02T05:33:03.340Z WARNING: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 4873 microseconds to 136102 microseconds.
2025-04-02T05:33:10.858Z WARNING: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 3498 microseconds to 143353 microseconds.
2025-04-02T05:33:32.061Z WARNING: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 3501 microseconds to 85882 microseconds.

Resolution

Engage the server vendor to investigate the device latency. Once fixed, the backups should not see CRC errors and the backups should complete successfully. 

Reference KB below

"performance has deteriorated" messages in ESXi host logs