Monitoring chart is not working for any Virtual Machine in Cloud Director 10.5

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Monitoring chart empty for all virtual machines(VM).
• Restarting the Cloud director cells did not resolve the issue.
• From the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, the following entries are seen with respect to the vcenter:

ERROR  | stats-activity-pool-26468 | StatsFeederCollectorActivity | [Activity Execution] Error retrieving performance stats from VC d12b3456-7fbc-8d90-1d23-4cacbcf5f67a in monitoring mode CASSANDRA - Handle: urn:uuid:efc5b6b4-e2a8-370b-8926-d92c2b4f81fd, Current Phase: StatsFeederCollectorActivity$StatsFeederCollectorPhase | activity=(com.vmware.vcloud.metrics.statsfeeder.StatsFeederCollectorActivity,urn:uuid:efc5b6b4-e2a8-370b-8926-d92c2b4f81fd)
(vmodl.fault.InvalidArgument) {
  faultCause = null,
  faultMessage = null,
  invalidProperty = entity

• References to a stale VM in VCD which is not present in vcenter is also seen in the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log:

DEBUG  | pool-jetty-81560     | VmManagerImpl         | Could not retrieve status for VM TestVM2 from the inventory
WARN  | pool-jetty-81560     | RclServiceImpl        | Cannot retrieve inventory record for VM TestVM2 | requestId=ecf123b4-fe56-8bc7-ab0e-b12f3a456cc7,request=GET https://vcd.example.com/api/vApp/vm-12345b6f-789a-1234-56fc-34abaf0c7a5d,requestTime=171234355545,remoteAddress=123.123.123.123:12345,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 39.0.0-alpha
com.vmware.ssdc.library.exceptions.InventoryObjectNotFoundException: Could not find object with moref "vm-1234" and VC ID "d12b3456-7fbc-8d90-1d23-4cacbcf5f67a" in inventory category "VirtualMachine".

 

Environment

VMware Cloud Director 10.x

Cause

This issue occurs due to a stale virtual machine in Cloud Director which do not exists in vCenter. When the stats couldn't be retrieved for the stale VM, the statsfeeder component in VCD which is responsible for collecting metrics for all the VMs errors out as seen in the log entries.

Resolution

To resolve this issue, 

1. Delete the stale VM from the Cloud Director as per the documentation.
   1. If the VM cannot be removed from the Cloud director UI, please open a Support Request by referencing this article. Database modifications are required to delete stale records, which will require VMware Technical Support assistance.
2. Reconnect the vCenter.