"Cannot import VM ... since it is already managed by VMware Cloud Director." error when performing a failover in Cloud Director Availability

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Performing a failover between two Organization VDCs, each backed by a different vCenter Server, within the same Cloud Director instance in Cloud Director Availability periodically fail with an error similar to:

Unexpected vCloud Director error. [ UI-c9725142-e110-40d5-84fd-2920cc38ae63-r625-SV-04-19978d8c-1a52-4623-90a9-b1720ebc245f ] Cannot import VM VM-Name-hIxB (vm-4785) since it is already managed by VMware Cloud Director.

• Reviewing the source VM via the Cloud Director API, you see that the associated vSphere moref values for this VM are now pointing to the recovered VM in the destination vCenter Server. This results in the source VM on the source vCenter Server no longer being managed by Cloud Director.
• In the /opt/vmware/h4/cloud/log/cloud.log file on the Cloud Replication Management Appliance, you see an entry similar to: 

2022-03-22 14:21:37.327  WARN - [UI-c9725142-e110-40d5-84fd-2920cc38ae63-r625-SV-04] [job-59] com.vmware.h4.cloud.job.VmFailoverJob    : Failed to import the recovered VC VM into a new vCD vApp

com.vmware.vcloud.client.exception.VcloudException: (Major code = 400, minor code = BAD_REQUEST) - [ UI-c9725142-e110-40d5-84fd-2920cc38ae63-r625-SV-04-246d69e8-55ad-44d0-bfa0-c0d543540847 ] Cannot import VM VM-Name-hIxB (vm-4785) since it is already managed by VMware Cloud Director.
        at com.vmware.vcloud.client.VcloudClient.lambda$defaultErrorDeserializer$0(VcloudClient.java:251)
        at com.vmware.rest.client.AbstractRestClient.convert(AbstractRestClient.java:225)
        at com.vmware.vcloud.client.XmlRestClient.exchange(XmlRestClient.java:92)
        ...

• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on the Cloud Director cell, you see entries similar to:

2022-03-22 14:21:35,549 | ERROR    | pool-jetty-73             | ImportServiceImpl              | Trying to import a VM VM-Name-TK-hIxB with moref vm-4785, but it is already managed by vCD | requestId=UI-c9725142-e110-40d5-84fd-2920cc38ae63-r625-SV-04-246d69e8-55ad-44d0-bfa0-c0d543540847,request=POST https://vcd-url/api/admin/extension/vimServer/e758358e-3448-4d35-935b-14accc51b73c/importVmAsVApp,requestTime=1647958895447,remoteAddress=172.22.88.20:42086,userAgent=Apache-HttpClient/4.5.13 (Java/11.0.12),accept=application/*+xml;version 33.0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x
VMware Cloud Director 10.x

Cause

This issue occurs when Cloud Director performs a VM identity check of the recovered VM before Cloud Director Availability requests the VM be imported to Cloud Director.

When performing the VM identity check, Cloud Director observes that the recovered VM is identical to the source VM but is placed on a different vCenter Server. This prompts Cloud Director to update its database repointing the source VM in Cloud Director to the recovered VM in vCenter Server causing the subsequent import request from Cloud Director Availability to fail.

Note: It can be observed that this issue does not occur on every failover attempt as sometimes Cloud Director Availability is able to import the recovered VM into Cloud Director before it can perform this VM identity check.

Resolution

This issue is resolved in Cloud Director 10.3.3.1

Additional Information

Once a failover has been attempted and you have encountered this issue, it is not possible to recover the original source VM configuration via Cloud Director Availability.

To fix the Cloud Director records for any affected VMs, contact VMware Cloud Director Support and note this Article ID (88147) in the problem description. For more information, see Creating and managing Broadcom support cases.