VM Data corruption or redo log errors after storage migration

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines (VMs) experience silent data corruption or filesystem inconsistencies during high-throughput I/O operations (e.g., Storage vMotion, backup, or SAN migration).
• VMs fail to power on and may present the error: The redo log of '####.vmdk' is corrupted

Environment

VMware vSphere ESXi (All versions)

Cause

HBA firmware and ESXi drivers are unsupported or misaligned as per Broadcom Compatibility Guide.

When HBA firmware and ESXi drivers are misaligned or unsupported, they might lose the ability to communicate predictably during intensive I/O events. This communication gap can cause:

1. Queue Command Desynchronization: Commands are processed out of order or dropped.
2. Packet Drops: Data packets fail to reach the storage array correctly.
3. Buffer Processing Errors: Internal HBA buffers overflow or desync, leading to corrupted blocks being silently written to the storage array without an immediate SCSI sense error.

Resolution

To resolve this issue and prevent silent data corruption, align the HBA driver and firmware to a supported combination.

1. Identify Current Versions:
   • Check the driver version: esxcli software vib get | grep <driver_name>
   • Check the firmware version: esxcli storage san fc list or check the HBA details in the vSphere Client.
2. Verify Compatibility:
   • Refer to the Broadcom Compatibility Guide to find the minimum required firmware for your specific driver and ESXi version.
3. Update HBA Firmware:
   • Follow the hardware vendor's instructions for applying firmware updates to your system.
4. Update Server BIOS:
   • Ensure the host BIOS is up to date, as HBA stability is often linked to the PCIe bus handling in the system BIOS.
5. Health Check:
   • After updating, perform a storage vendor health check to ensure no persistent FC fabric errors or CRC issues remain on the SAN switches.

IMPORTANT: Data already corrupted in the redo logs or VMFS blocks cannot be fixed by a firmware update. Affected VMs must have their filesystems repaired (e.g., using fsck or chkdsk) or be restored from a known good backup after the HBA alignment is complete.

Additional Information

The following factors should also be evaluated as potential sources of these symptoms:

Virtual machine disks become corrupted after storage vMotion between two different arrays