VAMI shows Cluster Health: Read Only Primary or Degraded state

Products

• VMware Cloud Director
• VMware Telco Cloud Platform

Issue/Introduction

• After a failed upgrade snapshot revert or increasing the capacity of the embedded PostgreSQL database on VMware Cloud Director Appliance by following the Increase PostgreSQL database disk , the Cluster Health shows DEGRADED or READ ONLY PRIMARY
  
  
  
  
  NOTE: '!' mark before the node means - standby node is detached from its upstream primary.
  
  
• Cannot access VMware Cloud Director UI, instead getting the below error when trying to access the page:
  
  HTTP ERROR 404 JSP file [/error.jsp] not found
  
  
• Upgrade process failed and now the service will not start
• NFS Server used for the Transfer Server Storage was in-accessible and now the VAMI shows "READ ONLY PRIMARY"

Environment

VCD 10.4.3, 10.x

TCP 5.0

Cause

• Cluster Health shows DEGRADED when the primary and one of the standby cells are online and operational, but the other standby cell is non-functional. The primary database is functional in this state, but if there is another database failure of either of the operational cells, the primary will become non-functional. The non-functional standby cell must be replaced with a new, functioning standby cell as soon as possible to restore the cluster to a Healthy state. The VMware Cloud Director UI and API are functional.
• Cluster Health shows READ ONLY PRIMARY when there is an online primary database, but it is READ_ONLY because the environment does not have an operational standby cell. Two new standby cells must be deployed. The VMware Cloud Director UI and API are not available.

Resolution

• To workaround the issue please use the following KB Article.

Reverting from a Cloud Director Appliance Deployment with Database HA to a Cloud Director Appliance Deployment Without Database HA

• If the above does not work please unregister the faulty Standby cells using the following document: Unregister a Standby Cell in Your VMware Cloud Director Database High Availability Cluster and make VAMI HEALTHY with only Primary cell.
• After removing the faulty standby cells successfully you can redeploy the cells again using the following document: Configure Your VMware Cloud Director Standby and Application Cells
• If the above workaround  do not help, please contact technical support and note this Knowledge Article ID (384987) in the problem description. For more information, see How to Submit a Support Request.

Additional Information

VAMI の Cluster Health 欄に Read Only Primary または Degraded と表示される

This issue also can happen if after reverting back to snapshots, the cells are not correctly started in sequence and waiting for each to come online, creating the database miscommunication. In this case, attempt first a full sequence reboot.