Tag associations are not removed from vCenter Server database when associated objects are removed or deleted

Products

• VMware vCenter Server

Issue/Introduction

This article will provide a method to remove the stale entries for the stale tag associations for the obsolete/deleted objects from vCenter database (vPostgres)

• Virtual Machine tag associations are not deleted from the vCenter database when a virtual machine is deleted or removed from vCenter.
• vCenter Server may encounter performance issue during API based access/operation or may report access timeouts
• Log/event monitoring tools may report vpxd log spew with ManagedObjectNotFound messages
• vpxd logs may rollover faster than expected due to increased logging 
• /var/log/vmware/vpxd/vpxd.log example below for various objects that could be affected

yyyy-mm-ddThh:mm:ss error vpxd[51116] [Originator@6876 sub=Authorize opID=########-####-####-####-########3595 Authz-4e] MoRef: vim. VirtualMachine:vm -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim.dvs.DistributedVirtualPortgroup:dvportgroup -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl.fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim. StoragePod:group-p<id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedobjectNotFound
yyyy-mm-ddThh:mm: ss error vpxd[60251] [Originator@6876 sub=Authorize opID=########-####-####-####-########7956 Authz-7f] MoRef: vim.ClusterComputeResource:domain-c<id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[7FB3F3AF6700] [Originator@6876 sub=Authorize opID=########-####-####-####-########976d Authz-4] MoRef: vim. HostSystem:host -< id> not found. Error: N5Vmomi5Fault21ManagedobjectNotFound9ExceptionE(vmodl. fault.ManagedObjectNotFound)
yyyy-mm-ddThh:mm: ss error vpxd[48210] [Originator@6876 sub=Authorize opID=########] MoRef: vim.Datastore:datastore -< id> not found. Error: N5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedobjectNotFound
yyyy-mm-ddThh:mm:ss error vpxd[11809] [Originator@6876 sub=Authorize opID=########-####-####-####-########ecb4 Authz-4f] MoRef: vim.Datacenter:datacenter -< id> not found. Error: N5Vmomi5Fault21ManagedobjectNotFound9ExceptionE(Fault cause: vmodl. fault.ManagedObjectNotFound

Note: The error "ManagedObjectNotFound" is a generic message for vCenter for failing to find an object. Messages pertaining to stale tag associations specifically are preceded by "Authz-##" 

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

• When a vSphere object (Virtual Machine, Host, Datastore, Distributed portgroup, Datacenter, Cluster, StoragePod) is deleted, the associated tag references are not deleted from cis_kv_keyvalue table of vCenter server database.
• With vSphere 7.x or later, the tags (created post upgrade/install) for Virtual Machine and Host only are saved under database table vpx_entity_tag_xref and are cleared with the deletion of the object unlike other objects.

Resolution

Note: Ensure to take a backup or a snapshot of the vCenter Server Appliance before performing this step. For ELM vCenters, review KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice and ensure offline snapshots of all vCenters in ELM are taken.

In order to clear the stale associations, proceed with the below steps

• Log in to the vCenter server appliance via ssh
• Copy the script tags_delete_job_all.sh to a persistent location on the vCenter server appliance via scp. If there is an issue connecting to a vCenter appliance using WinSCP, please see Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B
• Change the permission on the script to make the file executable

chmod +x tags_delete_job_all.sh

• Execute the script using 

  ./<directory>/tags_delete_job_all.sh

• If you get the below error while running the script: 

  bash: ./tags_delete_job_all.sh: /bin/bash^M: bad interpreter: No such file or directory

• This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor. To resolve this problem, run this command and rerun the script:

sed -i -e 's/\r$//' tags_delete_job_all.sh

 

Note: This script does not require a restart of vCenter Server services

Additional Information

The deleted information can be found under /var/log/vmware/vpxd-svcs/tagging_cron.log