Failover using VMware vCloud Director Availability fails with "Unexpected VMware Cloud Director error. [UI] Internal Server Error".

Products

• VMware Cloud Director

Issue/Introduction

• Attempts to perform a failover in vCloud director availability fails with  "Unexpected VMware Cloud Director error. [UI] Internal Server Error".
• The VM gets configured for the replication and synchronization also completes but the failover task fails.
• A number of Unresolved vApps piles up at the target VDC and does not get deleted.
• You see 'Task to import the recovered VC VM into a new vCD vApp has failed' error message in the cloud logs.

In /opt/vmware/h4/cloud/log/cloud.log on the VCDA Replication Manager, you see entries similar to:

2024-06-15 INFO - [UI] [job-] com.vmware.h4.cloud.job.VmFailoverJob : Manager task succeeded RecoveryInfo{recoveryState=failover, vcId='11111111-2222-3333-4444-555555555555', vmId='vm-ID', vmName='X', optimizeUntil=null}
2024-06-15  INFO - [UI] [job] com.vmware.h4.cloud.job.VmFailoverJob : Importing recovered VC VM into vCD.
2024-06-15  DEBUG - [UI] [job] com.vmware.h4.cloud.job.VmFailoverJob : Obtaining lock: vapp_C4VAPP-ID
2024-06-15  DEBUG - [UI] [job] c.vmware.h4.jobengine.lock.LockManager : Owner: X obtained lock: vapp_ID
2024-06-15 DEBUG - [UI] [job] com.vmware.h4.cloud.job.VmFailoverJob : Importing recovered vm into vCD: RecoveryInfo{recoveryState=failover, vcId='11111111-2222-3333-4444-555555555555'., vmId=''vm-ID', vmName='X', optimizeUntil=null}
2024-06-15 WARN - [UI] [task-poller-2] com.vmware.h4.cloud.job.VmFailoverJob : Task to import the recovered VC VM into a new vCD vApp has failed.
com.vmware.vcloud.client.exception.VcloudException: (Major code = 500, minor code = INTERNAL_SERVER_ERROR) - [ UI ]

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 

Environment

VMware vCloud Director Availability 4.6.1

Cause

This issue occurs when there are stale or missing entries in the vCD database.

Resolution

To resolve this issue, follow the below steps:

• Login to the vCloud director management interface.
• Navigate to vApp and select all the VMs with "UNRESOLVED" status and select "DELETE".
• Refresh the vCloud director and the vCenter server connection on the vCloud Director Availability management interface.
• Re-pair the VCDA sites and initiate the failover again. 

If the issue persists, raise a Support Request to proceed with advance troubleshooting of the issue.

Additional Information

To refresh the vCenter server address (Lookup address to the VCDA instances - Cloud, manager, tunnel and the replicator), please follow the instructions from vCloud director availability official document: https://docs.vmware.com/en/VMware-Cloud-Director-Availability/4.7/VMware-Cloud-Director-Availability-Install-Config-Upgrade-Cloud/GUID-E89D32A9-3B92-40EE-ABF4-3BEB2698A893.html?hWord=N4IghgNiBcIQ9vA1gVwA4AIwBNsCcBTAZyJAF8g