Error "Invalid configuration for device '##':" triggers failure of vMotion/Cloning for large Virtual Machines

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Shared-nothing vMotion (migration of both compute and storage) or cloning fails for very large Virtual Machines if the migration takes more than 24 hours to complete.

• Log entries similar to the following confirm the symptoms and the failure.
  • Destination vCenter Server logs (var/log/vmware/vpxd/vpxd.*log):
    YYYY-MM-DDTHH:MM:SS.MMMZ error vpxd[####] [Originator@6876 sub=VmProv opID=#######] Exception while executing action vpx.vmprov.CreateDestinationVm: N3Vim5Fault17InvalidDeviceSpec9ExceptionE(Fault cause: vim.fault.InvalidDeviceSpec)


  • Destination ESXi Server host management agent logs (var/run/log/hostd.log):
    YYYY-MM-DDTHH:MM:SS.MMMZ Er(163) Hostd####]: [Originator@6876 sub=######.vmx opID=######## sid=####user=vpxuser:<no user>] Device spec doesn't match up with dvport/dvpg configuration


  • Source vCenter Server logs (var/log/vmware/vpxd/vpxd.*log):
    YYYY-MM-DDTHH:MM:SS.836Z error vpxd[####] [Originator@6876 sub=VmProv opID=#######] Get exception while executing action vpx.vmprov.CreateDestinationVm:

            --> (vim.fault.InvalidDeviceSpec) {
            -->    property = "virtualDeviceSpec.device.backing",
            -->    deviceIndex = 118,
            -->    msg = "Invalid configuration for device '##'.",

• Cross vCenter vMotion may get the following error message: Failed to get DVS state in the restore phase from the source host <#.#.#.#>

Environment

VMware vCenter Server 8.x

VMware vCenter Server 7.x

Cause

• Failure of such migrations is likely due to a missing port on the destination ESXi host.
• For any migration, vCenter Server reserves a DVS port for 24 hours. After 24 hours, the port reservation expires and the port is deleted by the DvsMonitor, hence the Virtual Machine migrated to the destination host has no port to connect to and the migration/clone fails.

Resolution

To resolve this expected behavior, extend the port reservation timeout value on both the source and destination vCenter Servers to allow the migration to complete successfully.

1. SSH to the vCenter Server appliance.
   
   
2. Copy/backup the configuration file:

cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

     3. Insert the timeout parameter within the <vpxd> tags:

  <dvs>

      <vnicportbackings>

    <PortReserveTimeoutInMin>7200</PortReserveTimeoutInMin>

      </vnicportbackings>

  </dvs>

     4. Restart the vpxd service: service-control --restart vmware-vpxd

     5. Re-try the vMotion/Clone operation on the affected VMs.

     6. Once the Virtual Machines have been migrated, revert the vpxd.cfg file back to its original state:

cp /etc/vmware-vpx/vpxd.cfg.bak /etc/vmware-vpx/vpxd.cfg

     7. Restart the vpxd service: service-control --restart vmware-vpxd

Note: Value 7200  will extend port reservation timeout to 5 days (24*60*5=7200), and it should be enough to cover any time-demanding (slow/large VM (50TB+) Shared-nothing vMotion/Clone.