Stale snapshots in the datastore on which the virtual machine resides

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Stale snapshots such as, vmname-00000x.vmdk, are visible when browsing the datastore on which the virtual machine resides.

• These stale snapshots are not visible in Snapshot Manager.

• This issue may occur when a backup application is running,

 

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.0.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0

Cause

Third-party backup applications send API calls for VMware to create and remove snapshots on virtual machines they process. Snapshots may be left on virtual machines if those API calls are not sent, not received, or the snapshot removal operation cannot complete successfully.

If a snapshot remains after a third-party backup application has completed and that snapshot is visible in the Snapshot Manager, this may indicate that the snapshot removal API call was not sent or it was sent but was not received by the VMware environment. 

If a snapshot remains but is not visible in the Snapshot Manager, this indicates that the request for snapshot removal was sent and received. However, the snapshot consolidation was not completed successfully. This will often leave the virtual machine in a state where an alert of “Consolidation is needed." The snapshot removal process may be hindered by a lock on the VMDK files of that virtual machine, caused by the VMDK still be attached with Virtual Appliance transport mode or the VMDKs locked by the ESXi host.

See Consolidating/Committing snapshots in VMware ESXifor more information.
See Investigating virtual machine file locks on ESXi hosts for more information

Resolution

To resolve this issue:

To resolve this issue, verify if the virtual machine is running from the base snapshot and then delete the snapshot.

 

To confirm if the virtual machine is running from the snapshot or the base disk:

 

1. Right-click the affected virtual machine and click Edit Settings.

2. Select the hard disk and note the datastore in which it is located and check if it is pointing to the base disk (VM_Name.vmdk) or a snapshot disk (VM_Name-00000x.vmdk).

 

To delete the snapshot:

 

From the vSphere Client:

 

1. Right-click the virtual machine and click Snapshot.

2. Select Take Snapshot.

3. Enter a name for the snapshot and take a regular Snapshot.

4. After the snapshot is taken, right-click Virtual machine and click Snapshot > Snapshot Manager

5. Select the snapshot and click Delete All (Consolidate)

 

From ESXi command line:

 

1. Log in to the ESXi host as root using SSH.

2. Run this command to list all the virtual machines running on the host:

# vim-cmd vmsvc/getallvms

3. Run this command to list the snapshot for a particular virtual machine:

# vim-cmd vmsvc/snapshot.get VMID

Note: To determine the VMID, run the getallvms command.

 

4. Run this command to create a snapshot:

# vim-cmd vmsvc/snapshot.create VMID snapshot_Name Snapshot_Description Memory_Included Snapshot_Quiesced


Note: Do not close the SSH session during the process

5. After snapshot is created, run this command to commit the snapshot:

# vim-cmd vmsvc/snapshot.removeall VMID

 

Additional Information

Time required to remove the snapshot might be longer if the snapshot size is large.

This issue can occur even after migrating the virtual machine with snapshots using Storage vMotion

When a virtual machine with snapshots is migrated using Storage vMotion,  the old and the new datastore is referenced in the summary tab of the virtual machine. This is because the snapshot still has references to the old datastore.