Removing host(s) from SDDC Manager fails with error - Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory

Products

• VMware Cloud Foundation

Issue/Introduction

• When attempting to remove a host from an inactive cluster, it fails with the below error:
  
  Fetch Remove ESXi Host(s) Data from the SDDC Manager Inventory
  
  
• Below entries are seen in the /var/log/vmware/vcf/domainmanager/domainmanager.log:

YYYY-MM-DD HH:MM:SS DEBUG [vcf_dm,############4b2f,30c7] [c.v.v.h.a.RemoveHostEntityStatusActionUtil,dm-exec-##] Validate cluster status
YYYY-MM-DD HH:MM:SS ERROR [vcf_dm,############4b2f,30c7] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-##] [T3M45D] REMOVE_HOSTS_FROM_NOT_ACTIVE_CLUSTER_ERROR Removal for hosts [FQDN1.Domain.local, FQDN2.Domain.local, FQDN3.Domainlocal] from not active cluster is not allowed.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Removal for hosts [FQDN1.Domain.local, FQDN2.Domain.local, FQDN3.Domainlocal] from not active cluster is not allowed.

Environment

• VMware Cloud Foundation 4.5
• VMware Cloud Foundation 4.4.x
• VMware Cloud Foundation 4.0.x
• VMware Cloud Foundation 4.3.x
• VMware Cloud Foundation 5.0
• VMware Cloud Foundation 4.2

Cause

When the cluster is not in an ACTIVE state, the removal of hosts is not supported. For removing all the hosts in the cluster, delete cluster operation has to be performed.

Resolution

Workaround:

Note: Before proceeding with the below steps, it's recommended to take a backup of the database or make sure that there is valid snapshot taken.

1. If the Cluster is operational in vCenter and marked as ERROR in SDDC Manager, then follow the below steps:
   1. Take snapshot of SDDC Manager VM.
   2. ssh to SDDC Manager with vcf user and su to root
   3. List the cluster(s) which is / are in ERROR state in SDDC Manager platform database
      psql -h localhost -U postgres -d platform -c "\x" -c "select * from cluster where status='ERROR'"
      
      
   4. Mark the Cluster as ACTIVE
      psql -h localhost -U postgres -d platform -c "update cluster set status='ACTIVE' where id='<cluster id from the above step>'"
      
      
   5. Retry the host(s) removal task from SDDC Manager UI
      
      
2. If the Cluster is non-operational or not needed, then follow the below document to remove the cluster from SDDC Manager inventory and vCenter.
   • Delete a vSphere Cluster from a Workload Domain