Symptoms:
- Attempting to power on a VM in Cloud Director.
- The Organization VDC where the VM is located is backed by a Provider VDC with multiple resource pools.
- The Provider VDC resource pools are backed by ESXi host clusters which use different distributed switches for NSX networking.
- One of the Provider VDC's resource pools is at capacity with no more resources available, requiring the VM to migrate and power on in another Provider VDC resource pool.
- Disconnecting the VM's network in Cloud Director before powering on allows the VM to migrate and power on in the second cluster, the VM's network can be reattached in Cloud Director after power on successfully.
- The power on tasks fail with an error in the Cloud Director cell /opt/vmware/vcloud-director/logs/vcloud-container-debug.log similar to:
| ERROR | task-service-activity-pool-123 | FutureUtil | Failed to power on VM <VM_NAME>(com.vmware.vcloud.entity.vm:<VM_UUID>) | requestId=<REQUEST_UUID>,request=POST https://<VCD_URL>/api/vApp/vm-<VM_UUID>/power/action/powerOn,requestTime=<REQUEST_TIME>,remoteAddress=<CLIENT_ADDRESS>,userAgent=<USER_AGENT>,accept=application/*+xml;version 32.0 vcd=<VCD_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_UUID> )
com.vmware.ssdc.util.RelocateException: null
at com.vmware.vcloud.val.internal.impl.RelocateVmActivity$WaitForInventoryUpdatePhase.invoke(RelocateVmActivity.java:301)
at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vcloud.val.exceptions.TaskManagerException: Task failed, vc-id={<VCENTER_UUID>}, task-moref={ManagedObjectReference: type = Task, value =
<TASK_MOREF>, serverGuid = null}, error={Invalid configuration for device '<DEVICE_NUMBER>'.}
at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$WaitForInventoryUpdatePhase.invoke(AsynchronousTaskWaitActivity.java:279)
... 7 more
Caused by: com.vmware.ssdc.library.exceptions.VimInvalidDeviceSpecException: The operation could not be performed because the device spec is invalid. Invalid configuration for device '<DEVICE_NUMBER>'.
at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:432)
at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
... 8 more
Caused by: (vim.fault.InvalidDeviceSpec) {
faultCause = null,
faultMessage = null,
property = virtualDeviceSpec.device.backing,
deviceIndex = <DEVICE_NUMBER>
}