Snapshots are not visible in Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Snapshots created within Cloud Director are no longer visible.
• The snapshots are visible in the vCenter UI on VMs.
• On the VCD cell within the log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log it's observed that the snapshot was removed from inventory as follows:
  
  DEBUG    | pc-activity-pool-#       | StoredVirtualMachineListenerImpl | VCD snapshot (moref: snapshot-####) for VM VMVALRef: [vcId=########-####-####-####-########, moref=vm-##] not found in new inventory update, deleting it | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:########-####-####-####-########)
  
  
• The /opt/vmware/vcloud-director/logs/vcloud-container-debug.log also reports the listener failed to process events like follows:
   
ERROR    | pc-activity-pool-1        | EventManagerImpl               | Listener failed to process event=VirtualMachineSnapshotChangedEvent([vcId=########-####-####-####-########, moref=vm-##],[SnapshotInfo{name='<snapshotname>', valRef='[vcId=########-####-####-####-########, moref=snapshot-###]', parentMoref='null', vmMoref='vm-###', isCurrent=true, snapshotUid=###, quiesced=false, replaySupported=false, state='poweredOn', dateCreated=<date_time>}],[SnapshotInfo{name='<snapshot name>', valRef='[vcId=########-####-####-####-########, moref=snapshot-###]', parentMoref='null', vmMoref='vm-###', isCurrent=true, snapshotUid=###, quiesced=false, replaySupported=false, state='poweredOn', dateCreated=<date_time>}],[]) | activity=(com.vmware.vcloud.vimproxy.internal.impl.PCEventProcessingActivity,urn:uuid:########-####-####-####-########)
com.vmware.ssdc.library.exceptions.VimNotConnectedException: The operation failed because vCenter Server "<name>" is not connected.
        at com.vmware.vcloud.val.vlsi.internal.impl.VlsiClientUtil$3.run(VlsiClientUtil.java:120)
        at com.vmware.vcloud.val.vlsi.internal.impl.VlsiClientUtil$3.run(VlsiClientUtil.java:104)
  
  
• After reconnecting vCenter server, VM snapshots are not visible.

Environment

VMware Cloud Director 10.6.1.1

Cause

This issue can occur if the vCenter Server is in a disconnected state in Cloud Director. If the listener attempts to process change events during this time it will fail to process the VirtualMachineSnapshotChangedEvent event which results in the snapshot reference being removed from inventory.

Resolution

To resolve this issue set the vm.hideExternalSnapshots parameter to false and reconnect the vCenter. VCD will scan the vCenter objects during this process and it will resync the snapshots to it inventory.

1. Log in as root via SSH to one of the Cloud Director cells.
2. Run the following command to validate the current setting:
   
   cmt manage-config -n vm.hideExternalSnapshots -l
   
   Note: If the command response states "Property "vm.hideExternalSnapshots" does not exist" then this means the parameter is utilizing the default coded value and was not modified by the user before.
   
   
3. Run the following command to set vm.hideExternalSnapshots to false.
   
   cmt manage-config -n vm.hideExternalSnapshots -v false
   
   
4. Confirm the change is applied by running the command:
   
   cmt manage-config -n vm.hideExternalSnapshots -l
   
   
5. Access the Cloud Director Provider UI and perform the steps outlined in Reconnect a vCenter Instance to Your VMware Cloud Director.
6. Access the Tenant UI and confirm the snapshots are visible.
7. Revert the vm.hideExternalSnapshots to the default setting using the command:
   
   cmt manage-config -n vm.hideExternalSnapshots -d

 

If errors prevent the snapshots from being reintroduced successfully using a reconnect, as detailed above, removing the snapshot(s) from the underlying vCenter to align the configuration between vSphere and VMware Cloud Director (VCD) can work around the behavior and permit VCD to resume control of the snaphots on the virtual machine, depending upon the underlying issue.

Additional Information

Since Cloud Director 10.6.1.1 the parameter vm.hideExternalSnapshots can be used to prevent external snapshots from being counted against a tenant's Org VDC storage allocation. Refer to Tenant Storage Quota Increases Unexpectedly During Snapshot Operations in VCD 10.6.x