Virtual machines become unmanageable in VMware Cloud Director after being re-added to the vCenter Server Inventory

Products

• VMware Cloud Director

Issue/Introduction

• After re-adding a virtual machine (VM) back into the vCenter Server Inventory the virtual machine enters an Unknown State in VMware Cloud Director (VCD).
• The VM options are greyed out and the VM cannot be managed in the VMware Cloud Director Tenant Portal.
• In the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log file on the VMware Cloud Director cells, you see messages similar to:
  
  YYYY-MM-DD hh:mm:ss,### | WARN     | pool-jetty-26             | RclServiceImpl                 | Cannot retrieve inventory record for VM <vm_name> | requestId=########-####-####-####-########1699,request=GET https://vcd.example.com/api/vApp/vm-########-####-####-####-########dc2e,requestTime=#############,remoteAddress=##.##.##.##:####,userAgent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:153.0) Gecko/20100...,accept=application/*+xml;version 40.0.0-alpha
com.vmware.ssdc.library.exceptions.InventoryObjectNotFoundException: Could not find object with moref "vm-###7" and VC ID "########-####-####-####-########e821" in inventory category "VirtualMachine".
        at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.getInvRec(VC20VirtualServer.java:3884)
        at com.vmware.vcloud.val.internal.impl.VC20VirtualServer.getInvRec(VC20VirtualServer.java:3878)
        at com.vmware.vcloud.rcl.impl.RclServiceImpl.populateVimVmInfo(RclServiceImpl.java:1930)
        at com.vmware.vcloud.rcl.impl.RclServiceImpl.getVimVmInfoHelper(RclServiceImpl.java:1914)
        at com.vmware.vcloud.rcl.impl.RclServiceImpl.getVimVmInfo(RclServiceImpl.java:1902)
        ...

Environment

VMware Cloud Director 10.6.1.x

Cause

This issue occurs because the virtual machine Managed Object Reference (MORef) changes when added back to the vCenter Server Inventory while VMware Cloud Director still references the original MORef. This can happen when:

• A virtual machine was removed from the vCenter Server Inventory due to the virtual machine being in an orphaned state.
• A virtual machine was removed incorrectly and needed to be added back to the Inventory.

Resolution

To resolve this issue, remove the virtual machines from management in Cloud Director and then re-import the impacted virtual machines into VMware Cloud Director.

1. How to remove a virtual machine from Management in Cloud Director.
2. Re-import the impacted virtual machines into VMware Cloud Director.

If you are unable to re-import the virtual machines, contact Broadcom Support and note this Article ID (337644) in the problem description. For more information, see Creating and managing Broadcom cases.

Additional Information

For more information on how to import virtual machines into VMware Cloud Director, see Options to import VMs into VMware Cloud Director.