SDDC Manager fails to remove ESXi host with error "Failed to migrate vCenter Server Attached Tags" due to out-of-band inventory changes

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When attempting to remove or decommission an ESXi host from a cluster within VMware Cloud Foundation (VCF), the workflow fails with error
  
  failed to migrate vcf attached tags to vcenter server.
  
  
• In /var/log/vmware/vcf/domainmanager/domain-manager.log, entries similar to the following are observed
  
  WARN [vcf_dm,<UUID>,cabd] [c.v.e.s.c.s.a.BaseMigrateAttachedTagAssociationsAction] Failed to migrate attached tags for host <REDACTED_ID>. Reference Token: OEEI4D.
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : "{"errorCode":"MIGRATION_OF_VCENTER_ATTACHED_TAGS_FAILED","arguments":["hosts","<REDACTED_ID>"],"message":"Failed to migrate vCenter Server Attached Tags to VCF for resource of Type hosts, with ID <REDACTED_ID>.","referenceToken":"XXXX"}"
  
  
• The host have already been manually removed from the cluster via the vSphere Client (vCenter) prior to initiating the task in SDDC Manager.

Environment

VMware SDDC Manager 5.x

Cause

This issue occurs due to an out-of-band (OOB) operation. In a VCF environment, all lifecycle management operations (such as adding or removing hosts) must be performed through the SDDC Manager interface.
When a host is removed directly from vCenter, the SDDC Manager inventory database becomes desynchronized. During the removal workflow, SDDC Manager attempts to interact with the host and its associated metadata (tags) on the vCenter side; since the host is no longer present in the vCenter cluster context, the tag migration task fails with a 500 Internal Server Error.

Resolution

To resolve the database mismatch and clean up the stale host records in SDDC Manager, perform the following steps:

1. Database Cleanup: Follow the instructions in the following KB to manually remove the stale ESXi host entries from the SDDC Manager inventory database: Unable to remove ESXi host from VCF SDDC manager due to database mismatch | Steps to remove ESXi Hosts from the SDDC Manager DB (Broadcom KB 322842) (Note: Ensure a file-based backup of the SDDC Manager VM is taken before performing any manual DB edits.)

2. Validation: After the database cleanup, refresh the SDDC Manager UI and verify that the host is no longer associated with the workload domain or cluster.

Additional Information

To prevent such synchronization issues in the future, it is highly recommended to plan an upgrade to VMware Cloud Foundation 9. VCF 9 introduces an Enhanced Inventory Sync feature, which is designed to automatically reconcile and synchronize the SDDC Manager inventory with out-of-band changes made directly in vCenter Server.