VMDK redo log corrupted due to high storage I/O latency

Products

• VMware vCenter Server

Issue/Introduction

• Virtual machine failed to power on.
• In the vmware.log of virtual machine there is below similar error messages:
  
  In(05) vcpu-4 - Msg_Question:
In(05) vcpu-4 - [msg.hbacommon.corruptredo] The redo log of '#####-000001.vmdk' is corrupted. If the problem persists, discard the redo log.
  
  
• In the vmkernel.log of ESXi host there is below message indicate the storage I/O latency is very high:
  
  cpu34:2111128)WARNING: ScsiDeviceIO: 1513: Device naa.<lun-id> performance has deteriorated. I/O latency increased from average value of 6326 microseconds to 137725 microseconds.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

The latency of datastore where the virtual machine is located is extremely high. This can cause delays in writing I/O operations to the disk, ultimately resulting in the corruption of vmdk redo log.

Resolution

• Contact the storage and path vendor to check why the IO Latency is so high. More information is in KB Troubleshooting poor Storage Controller performance for VMware ESXi

• To workaround the issue, revert snapshot for the virtual machine. More information is in KB Corrupt redo log causes errors within the virtual machine while powering on ESXi
  
  Note: current data lost risk if revert to old snapshot.