Removing host(s) from SDDC Manager fails with error - Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

When attempting to remove a host from an inactive cluster, it fails with the error:

Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory

/var/log/vmware/vcf/domainmanager/domainmanager.log 

2023-03-03T11:04:47.878+0000 DEBUG [vcf_dm,8a93c1ccf9c94b2f,30c7] [c.v.v.h.a.RemoveHostEntityStatusActionUtil,dm-exec-19] Validate cluster status
2023-03-03T11:04:47.878+0000 ERROR [vcf_dm,8a93c1ccf9c94b2f,30c7] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-19] [T3M45D] REMOVE_HOSTS_FROM_NOT_ACTIVE_CLUSTER_ERROR Removal for hosts [vcf-st-111-25.w2g-qes-001.rainpole.local, vcf-st-111-26.w2g-qes-001.rainpole.local, vcf-st-111-27.w2g-qes-001.rainpole.local] from not active cluster is not allowed.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Removal for hosts [vcf-st-111-25.w2g-qes-001.rainpole.local, vcf-st-111-26.w2g-qes-001.rainpole.local, vcf-st-111-27.w2g-qes-001.rainpole.local] from not active cluster is not allowed.

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 4.4.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 5.0
VMware Cloud Foundation 4.2

Cause

When the cluster is in not in an ACTIVE state, the removal of hosts is not supported.

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:

1. If the Cluster is operational in vCenter and marked as ERROR in SDDC Manager then follow the below steps
   • Take snapshot of SDDC Manager VM.
   • ssh to SDDC Manager with vcf user and su to root
   • List the cluster(s) which is / are in ERROR state in SDDC Manager platform database
     • psql -h localhost -U postgres -d platform -c "\x" -c "select * from cluster where status='ERROR'"
   • Mark the Cluster as ACTIVE
     • psql -h localhost -U postgres -d platform -c "update cluster set status='ACTIVE' where id='<cluster id from the above step>'"
   • Retry the host(s) removal task from SDDC Manager UI
2. If the Cluster is non-operational or not needed then follow the below document to remove the cluster from SDDC Manager inventory and vCenter.
   • Delete a vSphere Cluster from a Workload Domain
3. ​​​​​​​If the Cluster is already deleted from vCenter and is in ERROR state in SDDC Manager inventory
   • Open a SR with VMware GSS to assist you with manually removing the hosts from SDDC Manager database.