Error: "Internal Server Error - Expected completed future" when attempting to take a virtual machine snapshot with quiesce enabled

Products

• VMware Cloud Director
• VMware Live Recovery

Issue/Introduction

• The Cloud Director (VCD) UI displays the error: "Internal Server Error - Expected completed future, but received future which is still in progress".
• The vCenter Server tasks console shows the error: "Another task is already in progress".
• Snapshot deletion issues when quiescence is enabled.
• The issue affects Windows VMs where quiesce is enabled during replication or snapshotting tasks.
• The VM has an SLA Profile in VMware Cloud Director Availability (VCDA) which has the quiesce option enabled.
• Snapshot removal on the VM in vCenter Server is failing and the task reports:
  
  The guest OS has reported an error during quiescing
  
  
• Within the virtual machines vmware.log file the following error is seen:
  

  In(05) vmx - Msg_Post: Warning
In(05) vmx - [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
In(05)+ vmx - The error code was: 3
In(05)+ vmx - The error message was: Quiesce operation canceled
In(05) vmx - ----------------------------------------
In(05) vmx - ToolsBackup: changing quiesce state: FINISHING -> ERROR_WAIT
In(05) vmx - ToolsBackup: changing quiesce state: ERROR_WAIT -> IDLE

  In(05) vcpu-0 - SnapshotVMXTakeSnapshotComplete: Done with snapshot 'vSphere Replication quiesced snapshot': ##
In(05) vcpu-0 - SnapshotVMXTakeSnapshotComplete: Snapshot ## failed: Failed to quiesce the virtual machine (##).
In(05) vcpu-0 - SnapshotVMXTakeSnapshotComplete: Cleaning up incomplete snapshot ##.
In(05) vcpu-0 - SnapshotVMXTakeSnapshotComplete: Starting snapshot consolidation for cleaning up incomplete snapshots.
In(05) vcpu-0 - SnapshotVMX_Consolidate: Starting online snapshot consolidate operation

Environment

VMware Cloud Director 10.6.1.x
VMware Cloud Director Availability 4.7.x

Cause

This issue is caused by a failure within the Microsoft Volume Shadow Copy Service (VSS) subsystem or a specific VSS writer inside the Windows Guest OS. 

When VCD initiates a snapshot with quiesce enabled, it waits for a response from the Guest OS via VMware Tools.  If the VSS freeze window fails or takes longer than the default 5-minute VCD timeout, VCD reports an 'Expected completed future' error because it assumes the task has stalled. 

Resolution

To resolve this issue investigate within the Windows Guest OS to identify and resolve any problems with Microsoft VSS which would cause quiesce to fail. If is recommended that Microsoft Support be contacted to analyze the VSS related issues if required.

Recommended actions to begin the review within the Windows OS include:

1. Review Windows Event Logs (Application/System) for VSS errors.
2. Verify the status of the VSS Writer using VssAdmin (vssadmin list writers).
3. Ensure VMware Tools is up to date.

Workaround

To workaround the issue, disable quiesce in the VCDA SLA Profile.

1. Login to the VCDA Provider UI.
2. Navigate to Configuration > SLA Profiles.
3. Click on radio button to the left of the relevant SLA Profile.
4. Select the Edit button.
5. Move the Activate quiesce slider to the disabled position.
6. Click Apply to save the change.

Additional Information

To prevent premature timeouts for long-running VSS operations, the Cloud Provider can increase the global timeout settings using the cell-management-tool on the VCD cells.

1. To increase the snapshot task timeout. (Default is 300 seconds)
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vm.snapshotRevertTimeoutInSeconds -v 600
2. To increase the Inventory Update Timeout (Default is 300 seconds)
   
   /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n vcloud.val.inventory.updateTimeout -v 480