Unable to reconfigure VM during a cloud site test failover in Cloud Director Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

• When performing a test failover to a cloud site the task fails and you see similar messages in the Replication Tasks view in the Cloud Director Availability Portal:

Failed to connect NICs for vm: #######-####-####-####-###########

Unexpected vCloud Director error. [ UI-####-####-########-####-####-############-####-######## ] null

Unable to restore the Guest Customization settings of recovery virtual machine 'vm_name'.
Unexpected vCloud Director error.  [ UI-####-####-########-####-####-############-####-######## ] null

• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file on the Cloud Director cell, you see entries similar to:

ERROR | storage-fabric-activity-pool-#### | RelocateStoredVmByDatastoreActivity | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByDatastoreActivity$MigratePoweredOnVmAcrossRpAndDsPhase@##### - Handle: urn:uuid:#######-####-####-####-###########, Current Phase: RelocateStoredVmByDatastoreActivity$MigratePoweredOnVmAcrossRpAndDsPhase | requestId=ui####-####-########-####-####-############-####-######## ,request=PUT https://vcd_url/api/vApp/vm-############-#####-####-####-############/networkConnectionSection,requestTime=############,remoteAddress=X.X.X.X:Port,userAgent=Apache-HttpClient/4.5.12 (Java/11.0.7),accept=application/*+xml;version 31.0 vcd=############-#####-####-####-############,task=#######-####-####-####-########### activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:##########-####-####-####-##########) activity=(com.vmware.ssdc.backend.services.impl.RelocateVmActivity,urn:uuid:#######-####-####-####-###########) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByStorageClassActivity,urn:uuid:############-#####-####-####-############) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByDatastoreActivity,urn:uuid:############-#####-####-####-############)

java.util.concurrent.ExecutionException: com.vmware.ssdc.util.RelocateException: null
       at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:###)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:###)
       at java.util.concurrent.FutureTask.run(FutureTask.java:###)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:####)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:###)
       at java.lang.Thread.run(Thread.java:###)
Caused by: com.vmware.ssdc.util.RelocateException: null
       at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.relocateVm(VC20VirtualServer.java:####)
       at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.relocate(VC20VirtualServer.java:####)
       at com.vmware.vcloud.fabric.storage.storedVm.impl.RelocateStoredVmByDatastoreActivity$MigratePoweredOnVmAcrossRpAndDsPhase.baseInvoke(RelocateStoredVmByDatastoreActivity.java:####)

       ...

ERROR |storage-fabric-activity-pool-303 | CreateFromVCenterVmActivity    | [Activity Execution] Uncaught Exception during Activity execution. Recent phase: com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromVCenterVmActivity$ImportFromVcPhase@127539ee - Handle: urn:uuid:############-#####-####-####-############, Current Phase: CreateFromVCenterVmActivity$ImportFromVcPhase |requestId=UI-#######-####-####-####-###########-kZ-Tn-Ms2-#######-####-####-####-###########,request=POST https://vcd_url/api/admin/extension/vimServer/############-#####-####-############/importVmIntoExistingVApp,requestTime=#############,remoteAddress=X.X.X.X:Port,userAgent=Apache-HttpClient/4.5.12 (Java/11.0.7),accept=application/*+xml;version 31.0 vcd=############-#####-####-####-############,task=############-#####-####-####-############ activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:############-#####-####-####-############) activity=(com.vmware.vcloud.vdc.impl.ImportVmIntoExistingVAppActivity,urn:uuid:############-#####-####-####-############) activity=(com.vmware.vcloud.vdc.impl.CreateStoredVmActivity,urn:uuid:#######-####-####-####-###########) activity=(com.vmware.vcloud.fabric.storage.storedVm.impl.CreateFromVCenterVmActivity,urn:uuid:############-#####-####-####-############)

java.util.concurrent.ExecutionException: com.vmware.vcloud.val.exceptions.VimRemoteException: (vim.fault.MethodDisabled)

{
   faultCause = null,
   faultMessage = null,
   source = H4-############-#####-####-####-############
}
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:###)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:###)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:####)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:###)
        at java.lang.Thread.run(Thread.java:###)
Caused by: com.vmware.vcloud.val.exceptions.VimRemoteException:(vim.fault.MethodDisabled)

{
   faultCause = null,
   faultMessage = null,
   source = H4-############-#####-####-####-############
}
        at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.relocateVm(VC20VirtualServer.java:####)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.relocate(VC20VirtualServer.java:####)
        ...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue occurs because Cloud Director attempts to relocate a test failover VM during import or while performing a post failover reconfiguration. This task fails as the relocate method is disabled for test VMs by Cloud Director Availability.

The relocate task can be initiated when the placement algorithm in Cloud Director determines the test VM would be better placed on a different resource to the one it was on when initiating the import from vCenter Server to Cloud Director despite the relocate method being disabled. This can occur when:

• The Provider VDC of the target Organization VDC is backed by multiple resource pools in vCenter Server.
• The target Storage Policy is backed by an SDRS enabled datastore cluster in vCenter Server.

Resolution

This is a known issue affecting Cloud Director Availability 4.x.
Currently, there is no resolution.

Workaround:
To work around this issue:

• Ensure that the target Storage Policy for a replication is accessible only by a single Provider VDC resource pool.
• Ensure that SDRS is disabled when using datastore clusters in the destination site.