Unable to power on a virtual machine after cancelling snapshot consolidation

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Snapshot Consolidation was started when the Virtual Machine is powered off.
   
• VM is unable to power on the VM after cancelling the snapshot consolidation.
   
• When power on is attempted, it fails with an error message similar to:
  
  Failed to open '/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk' with flags 0x13 The system cannot find the file specified (25).
   
• In the /var/log/hostd.log file, similar entries are seen:
  
  2017-08-04T13:23:53.862Z info hostd[29EC2B70] [Originator@6876 sub=Libs opID=c4c8aa4f-da14-4209-a243-5e687f786047-162836-ngc-a1-72-1b7d user=vpxuser:Domain\User] SnapshotVigorConsolidate: Failed to consolidate: Operation was canceled (5)
  .
  .
  2017-08-04T13:23:53.868Z info hostd[28EC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 3340 : Virtual machine Virtual_Machine_Name disks consolidation failed on Cluster_Name in cluster Cluster_Name. in ha-datacenter.
  2017-08-04T13:23:53.885Z info hostd[28EC2B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 3341 : Virtual machine Virtual_Machine_Name disks consolidation is needed on Cluster_Name in cluster Cluster_Name. in ha-datacenter.
  2017-08-04T13:23:53.885Z info hostd[28EC2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmx] Send config update invoked
  2017-08-04T13:23:53.947Z info hostd[28EC2B70] [Originator@6876 sub=Libs] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk' : Could not find the file (393218).
  2017-08-04T13:23:53.947Z info hostd[28EC2B70] [Originator@6876 sub=DiskLib] DISKLIB-DSCPTR: DescriptorOpenInt: failed to open '/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk': Could not find the file (60002)
  2017-08-04T13:23:53.947Z info hostd[28EC2B70] [Originator@6876 sub=DiskLib] DISKLIB-LINK : "/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk" : failed to open (The system cannot find the file specified).
  2017-08-04T13:23:53.947Z info hostd[28EC2B70] [Originator@6876 sub=DiskLib] DISKLIB-CHAIN : "/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk" : failed to open (The system cannot find the file specified).
  2017-08-04T13:23:53.947Z info hostd[28EC2B70] [Originator@6876 sub=DiskLib] DISKLIB-LIB : Failed to open '/vmfs/volumes/datastore/Virtual_Machine_Folder/Virtual_Machine.vmdk' with flags 0x13 The system cannot find the file specified (25).

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Some older versions of the VMware ESXi software allow snapshot virtual disk consolidation and snaphot deletion processes to be cancelled. This is a known issue affecting VMware ESXi 6.0 and ESXi 6.5 .

These processes should not be cancelled at any time. Cancelling these processes has a very high chance of causing virtual disk corruption.

Resolution

To resolve the issue follow the steps below:

1. Restore the VM from a backup which was taken before the consolidation attempt
2. Update ESXi to a version which works to prevent consolidations from being cancelled
3. Despite the updates, the user interface may still allow an attempt to cancel these processes. Please refrain from doing so.

 The issue is resolved in the following versions:

-VMware ESXi 6.0 Patch 6 (Build:6921384)
-VMware ESxi 6.5 U1(Build:5310538)

Workaround:
To workaround the issue Repoint the VM to a parent virtual disk.

If the parent virtual disk has not been corrupted, this will allow the VM to power on.

Warning: This will cause all the data in the snapshot virtual disks to be lost. For instance, if the snapshot disk was in place for two months, two months' worth of data will be lost. Ensure having good backups of the virtual machine before taking this step.

Process to repoint the VM to a parent virtual disk:

1. Open the virtual machine vmx file and find the "scsi disk" which failed to open.
   1. Open SSH session to host were VM resides.
   2. Change directory to VM location : cd /vmfs/volumes/datastoreName/VMname/
   3. vi Vmname.vmx
   4. Look for (Example : scsi0:0.fileName = "TestVM-000008.vmdk")- search for the "scsi disk" mentioned in error message.
2. Update the entry with the name of its parent disk.
   • This is found from the name of disk and its numbering "TestVM-000007.vmdk".
3. Power on the virtual machine.
4. If the VM still does not power on, Repeat the procedure for the parent snapshot disk, and re-try the power on again.

Note: It is common for multiple snapshot virtual disks to become corrupted in these cases. The procedure may need to be retried several times until the virtual disk is found in the chain which will allow the VM to power on.

Additional Information

Failed to power on virtual machine

Impact/Risks:
Repointing a virtual machine to use a parent virtual disk will cause the data contained in the child virtual disk to be lost.