Tag associations are not removed from vCenter Server database when associated objects are removed or deleted

Products

• VMware vCenter Server

Issue/Introduction

This article will provide a method to remove the stale entries for the stale tag associations for the obsolete/deleted objects from vCenter database (vPostgres)

Symptoms:

• Virtual Machine tag associations are not deleted from the vCenter database when a virtual machine is deleted or removed from vCenter.
• vCenter Server may encounter performance issue during API based access/operation or may report access timeouts
• Log/event monitoring tools may report vpxd log spew with ManagedObjectNotFound messages
• vpxd logs may rollover faster than expected due to increased logging 
• /var/log/vmware/vpxd/vpxd.log

yyyy-mm-ddThh:mm:ss error vpxd[51116] [Originator@6876 sub=Authorize opID=########-####-####-####-########3595 Authz-4e] MoRef: vim. VirtualMachine:vm -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim.dvs.DistributedVirtualPortgroup:dvportgroup -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl.fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim. StoragePod:group-p<id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedobjectNotFound
yyyy-mm-ddThh:mm: ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim.ClusterComputeResource:domain-c<id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[7FB3F3AF6700] [Originator@6876 sub=Authorize opID=########-####-####-####-########976d Authz-4] MoRef: vim. HostSystem:host -< id> not found. Error: N5Vmomi5Fault21ManagedobjectNotFound9ExceptionE(vmodl. fault.ManagedObjectNotFound)
yyyy-mm-ddThh:mm: ss error vpxd[48210] [Originator@6876 sub=Authorize opID=########] MoRef: vim.Datastore:datastore -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedobjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[11809] [Originator@6876 sub=Authorize opID=########-####-####-####-########ecb4 Authz-4f] MoRef: vim.Datacenter:datacenter -< id> not found. Error: N5Vmomi5Fault21ManagedobjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

When a vSphere object (Virtual Machine, Host, Datastore, Distributed portgroup, Datacenter, Cluster, StoragePod) is deleted, the associated tag references are not deleted from cis_kv_keyvalue table of vCenter server database. With vSphere 7.x or later, the tags (created post upgrade/install) for Virtual Machine and Host only are saved under database table vpx_entity_tag_xref and are cleared with the deletion of the object unlike other objects.

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:

Note: Ensure to take a backup or a snapshot of the vCenter Server Appliance before performing this step.

In order to clear the stale associations, proceed with the below steps

• Log in to the vCenter server appliance via ssh
• Copy the script tags_delete_job_all.sh to a persistent location on the vCenter server appliance via scp. If you have troubles connecting to a vCenter appliance using WinSCP, please see Error when uploading files to vCenter Server Appliance using WinSCP
• Change the permission on the script to make the file executable

chmod +x tags_delete_job_all.sh

• Execute the script using 

  ./<directory>/tags_delete_job_all.sh

Note: This script does not require a restart of vCenter Server services

Additional Information

The deleted information can be found under /var/log/vmware/vpxd-svcs/tagging_cron.log