Powering on a virtual machine fails with the error: Cannot open the disk caused by corrupt redo log

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Powering on a virtual machine fails
• The hostd.log file contains the errors:
  • Cannot open the disk <path to snapshot vmdk> or one of the snapshot disks it depends on. Reason: Broken pipe.
  • Error code -57

Note: The hostd.log file is located at:

• ESX/ESXi 4.x – /var/log/vmware/
• ESXi 5.x onwards – /var/log/

 

• A Veeam backup fails with the error:
  
  unable to take snapshot of the VM. A previous snapshot exists.
  
  
• The vmware.log file (located in the folder of the virtual machine) contains entries similar to:
   
  Nov 20 22:03:36.487: vcpu-1| Msg_Question:
  Nov 20 22:03:36.489: vcpu-1| [msg.hbacommon.corruptredo] The RedoLog of Example-000002.vmdk has been detected to be corrupt. The
  Nov 20 22:03:36.489: vcpu-1| virtual machine needs to be powered off. If the problem
  Nov 20 22:03:36.489: vcpu-1| still persists, you need to discard the redolog
  Nov 20 22:03:36.489: vcpu-1| ----------------------------------------
  Nov 20 22:12:26.280: vcpu-1| Msg_Question: msg.hbacommon.corruptredo reply=0
  Nov 20 22:12:26.284: vcpu-1| Exiting because of failed disk operation.
  
  
• Committing a snapshot fails with an unknown error and the virtual machine powers off.
• If you run # vmkfstools -t10 -v 10 <path to the snapshot vmdk> to check the integrity of the snapshots, you see these errors in the output:
  
  DISKLIB-CHAINESX : ChainESXOpenSubChainNode: can't create deltadisk node Example-000002-delta.vmdk failed with error Broken pipe
  DISKLIB-CHAIN : "FSExchVM.1_4-000002.vmdk" : failed to open (Broken pipe).
  DISKLIB-LIB : Failed to open 'Example-000002.vmdk' with flags 0xe (Broken pipe).
  Failed to open virtual disk: Broken pipe 2097161
  AIOMGR-S : stat o=6 r=12 w=0 i=3 br=153600 bw=0

Environment

VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESX 4.1.x
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Embedded
VMware ESX Server 3.5.x
VMware ESXi 3.5.x Installable

Resolution

 

 

The vmware.log file entries indicate that the redo log is corrupt, which prevents the virtual machine from powering on.

 

 

 

NOTE: Some knowledge of how VMware snapshots would be needed in order to carry out the resolution in this KB, if in any doubt then open a support request with Broadcom VMware.

 

To resolve the issue:

1. Edit the .vmx file and point the virtual machine back to a previous snapshot. If a previous snapshot does not exist, point it back to the base virtual disk. Note that all data in the corrupted snapshot disks will be lost when pointing to the previous disk.
   
   To edit the .vmx file:
   1. Run the command:
      
      # vi </vmfs/volumes/vm_datastore/restore/Example.vmx>
      
      Where </vmfs/volumes/vm_datastore/restore/Example.vmx> is the path to your .vmx file
      
      
   2. Edit the field scsi0:0.fileName = "Example-000001.vmdk" to use a previous snapshot.
      
      
2. If you are committing all snapshots, rename the .vmsd file to clear the corrupted snapshot information.
   
   Note: Only renaming the .vmsd file if you are committing all snapshots. If reverting to a previous snapshot, edit the .vmsd file.
   
   
3. Take a new snapshot.
4. Delete all snapshots.

Additional Information

Consolidating/Committing snapshots in VMware ESXi