vSphere Tags are Unexpectedly Deleted by VMware Aria Automation when vCenter is in Enhanced Linked Mode (ELM)

Products

• VCF Automation

Issue/Introduction

In VMware Aria Automation (formerly vRealize Automation), vSphere tags may be deleted unexpectedly from vCenter Server when a Virtual Machine (VM) is deleted. This issue specifically occurs when multiple vCenter Servers are configured in Enhanced Linked Mode (ELM).

• In vCenter vpxd.log under /var/log/vmware/vpxd/, the following events can be seen : 

INFO com.vmware.sync.SyncMetadataUtils opId=] Checking if item can be deleted: urn:vmomi:InventoryServiceTag:xxxxxx-xxx-xxx-xxx-xxxxxxx:GLOBAL
INFO com.vmware.sync.SyncMetadataUtils opId=] Deleting metadata for item urn:vmomi:InventoryServiceTag:xxxxxx-xxx-xxx-xxx-xxxxxxx:GLOBAL

INFO com.vmware.cis.lotus.LdapUtils opId=] Deleted rdn cn=urn:vmomi:InventoryServiceTag:xxxxxx-xxx-xxx-xxx-xxxxxxx:GLOBAL

Environment

Product: VMware Aria Automation (All Versions)
Infrastructure: vCenter Servers configured in Enhanced Linked Mode (ELM)

Cause

While the vSphere Adapter is designed to only delete a tag definition if it is no longer associated with any other object, the tagging API used by the adapter is limited to the scope of the specific vCenter Server being queried. In an ELM environment, the adapter cannot natively detect if a tag is still in use on a different, linked vCenter Server. As a result, if a VM is deleted and it was the last object using a specific tag on that vCenter, VMware Aria Automation may delete the tag definition entirely, even if other VMs in the linked environment are still using it.

The vCenter tagging API provides usage information only for the specific vCenter host where the API invocation occurs. To accurately determine if a tag is "in use" across an ELM setup, a client would need to invoke APIs across every linked vCenter. Currently, the VMware Aria Automation deletion function only queries the local vCenter before removing what it perceives to be a "stale" tag. This is corner case where the very first VM creation with a new TAG fails at certain level of the VM creation and hence cleanup is initiated. 

Resolution

A potential future update to the product is under consideration to utilize APIs that identify if a vCenter is linked. In such a scenario, VMware Aria Automation would skip the tag deletion process for linked vCenters while maintaining existing functionality for standalone instances.

As workaround, one of the following methods can be followed :

• Method 1: Persistent Tag Association
  Create a "dummy" or placeholder VM on each vCenter Server within the ELM group and assign the required tags to these VMs. Because the adapter will detect that the tag is still associated with the placeholder VM, it will not trigger the deletion of the tag definition when other managed VMs are removed.
• Method 2: Restrict vCenter Permissions
  You can prevent VMware Aria Automation from deleting tags by removing the specific privilege from the service account used for the vCenter Cloud Account.

Steps to Remove Delete Tag Privilege:

1. 1. Log in to the vCenter Server UI.
   2. Navigate to Administration > Roles.
   3. Select the Role assigned to the VMware Aria Automation Cloud Integration/Service Account.
   4. Click Edit to modify the role's privileges.
   5. Locate and deselect the Delete vSphere Tag privilege.
   6. Click Save to apply the changes.