Removing host(s) from SDDC Manager fails with error - Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

When attempting to remove a host from an inactive cluster, it fails with the error:

Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory

/var/log/vmware/vcf/domainmanager/domainmanager.log 

YYYY-MM-DD HH:MM:SS DEBUG [vcf_dm,xxxxxxxxxxxx4b2f,30c7] [c.v.v.h.a.RemoveHostEntityStatusActionUtil,dm-exec-19] Validate cluster status
YYYY-MM-DD HH:MM:SS ERROR [vcf_dm,xxxxxxxxxxxx4b2f,30c7] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-19] [T3M45D] REMOVE_HOSTS_FROM_NOT_ACTIVE_CLUSTER_ERROR Removal for hosts [FQDN1.Domain.local, FQDN2.Domain.local, FQDN3.Domainlocal] from not active cluster is not allowed.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Removal for hosts [FQDN1.Domain.local, FQDN2.Domain.local, FQDN3.Domainlocal] from not active cluster is not allowed.

 

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 4.4.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 5.0
VMware Cloud Foundation 4.2

Cause

When the cluster is in not in an ACTIVE state, the removal of hosts is not supported. For removing all the hosts in the cluster, delete cluster operation has to be performed.

Resolution

Workaround:

1. If the Cluster is operational in vCenter and marked as ERROR in SDDC Manager then follow the below steps
   • Take snapshot of SDDC Manager VM.
   • ssh to SDDC Manager with vcf user and su to root
   • List the cluster(s) which is / are in ERROR state in SDDC Manager platform database
     • psql -h localhost -U postgres -d platform -c "\x" -c "select * from cluster where status='ERROR'"
   • Mark the Cluster as ACTIVE
     • psql -h localhost -U postgres -d platform -c "update cluster set status='ACTIVE' where id='<cluster id from the above step>'"
   • Retry the host(s) removal task from SDDC Manager UI
2. If the Cluster is non-operational or not needed then follow the below document to remove the cluster from SDDC Manager inventory and vCenter.
   • Delete a vSphere Cluster from a Workload Domain

Note: Before proceeding further with the above steps its recommended to take a backup of the database or make sure that there is valid snapshot taken.