"The operation could not be performed because the device spec is invalid. Invalid configuration for device 'DEVICE_NUMBER'" error causes VM power on to fail in Cloud Director when ESXi cluster migration is required

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Attempting to power on a VM in Cloud Director.
• The Organization VDC where the VM is located is backed by a Provider VDC with multiple resource pools.
• The Provider VDC resource pools are backed by ESXi host clusters which use different distributed switches for NSX networking.
• One of the Provider VDC's resource pools is at capacity with no more resources available, requiring the VM to migrate and power on in another Provider VDC resource pool.
• Disconnecting the VM's network in Cloud Director before powering on allows the VM to migrate and power on in the second cluster, the VM's network can be reattached in Cloud Director after power on successfully.
• The power on tasks fail with an error in the Cloud Director cell /opt/vmware/vcloud-director/logs/vcloud-container-debug.log similar to:

| ERROR | task-service-activity-pool-123 | FutureUtil           | Failed to power on VM <VM_NAME>(com.vmware.vcloud.entity.vm:<VM_UUID>) | requestId=<REQUEST_UUID>,request=POST https://<VCD_URL>/api/vApp/vm-<VM_UUID>/power/action/powerOn,requestTime=<REQUEST_TIME>,remoteAddress=<CLIENT_ADDRESS>,userAgent=<USER_AGENT>,accept=application/*+xml;version 32.0 vcd=<VCD_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_UUID> )
com.vmware.ssdc.util.RelocateException: null
   at com.vmware.vcloud.val.internal.impl.RelocateVmActivity$WaitForInventoryUpdatePhase.invoke(RelocateVmActivity.java:301)
   at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
   at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.vmware.vcloud.val.exceptions.TaskManagerException: Task failed, vc-id={<VCENTER_UUID>}, task-moref={ManagedObjectReference: type = Task, value =
 <TASK_MOREF>, serverGuid = null}, error={Invalid configuration for device '<DEVICE_NUMBER>'.}
   at com.vmware.vcloud.val.taskmanagement.AsynchronousTaskWaitActivity$WaitForInventoryUpdatePhase.invoke(AsynchronousTaskWaitActivity.java:279)
  ... 7 more
Caused by: com.vmware.ssdc.library.exceptions.VimInvalidDeviceSpecException: The operation could not be performed because the device spec is invalid. Invalid configuration for device '<DEVICE_NUMBER>'.
   at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:432)
   at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:499)
   ... 8 more
Caused by: (vim.fault.InvalidDeviceSpec) {
 faultCause = null,
 faultMessage = null,
 property = virtualDeviceSpec.device.backing,
 deviceIndex = <DEVICE_NUMBER>
}

Environment

VMware Cloud Director 10.x

Cause

This is a known issue where migrating and powering on Cloud Director VMs will fail if the Provider VDC resource pools are backed by ESXi host clusters which use different distributed switches for NSX networking.

Resolution

This issue is resolved in Cloud Director 10.5, 10.4.2, and 10.3.3.4 which are available here, Download VMware Cloud Director.

Workaround:

To workaround the issue if an upgrade is not possible, detach the VM's network in Cloud Director before powering on and reattach it after the power on succeeds.

Example steps would be as follows:

1. In the Cloud Director Tenant portal open the VM's network configuration, see Edit Virtual Machine Properties in the VMware Cloud Director Tenant Portal.
2. Deselect the Connected check box to disconnect the NICs.
3. Power on the VM and wait for the task to complete.
4. After the VM has powered on, open the VM's network configuration and select the Connected check box to reconnect the NICs.