Error: Cluster is in ERROR status and it's not ACTIVE in SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Software-Defined Data Center (SDDC) Manager reports a cluster status as ERROR. When you view the cluster summary page, an error notification appears.

• In SDDC /var/log/vmware/vcf/domainmanager/domainmanager.log, below entries are observed.
  
  [YYYY-MM-DD-THH:MM:SS]  DEBUG [vcf_dm,0000000000000000,0000] [c.v.v.c.service.ResourceCacheService,ForkJoinPool.commonPool-worker-472]  Cluster ########-####-####-####-############ is in ERROR status and it's not ACTIVE. Skip adding this resource to the inventory.
  
  

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

The cluster status can enter an ERROR state when a configuration modification is performed directly outside the SDDC Manager or cluster upgrade performed which failed leaving cluster in stale state.

Resolution

1. Access the SDDC Manager appliance through SSH using the vcf user credentials.
2. Elevate to the root user.
3. Connect to the SDDC Manager database:
   psql -U postgres -h localhost -d platform
4. Verify the current status of the affected cluster:
   select id, status, name from cluster where id = '###'; Replace #### with the cluster ID identified in the domainmanager.log error.

If the cluster is reported in an ERROR state, open a Technical Support case with Broadcom and provide the query output for further investigation.