Error: "Cluster Health: CRITICAL PROBLEM" for Primary Cell Embedded Database Availability in VAMI UI and Cloud Director UI unavailable

Products

• VMware Cloud Director

Issue/Introduction

• Cloud Director UI and API are unavailable.
• The Embedded Database Availability in the Cloud Director VAMI UI for the Primary Cell shows statuses of:
  
  Cluster Health: CRITICAL PROBLEM

Failover Mode: INDETERMINATE
  
  
• The vpostgres.service on the Primary Cell is stopped:
  
  systemctl status vpostgres.service
...
Active: failed (Result: exit-code)
...
postgres [###]: pg_ctl: could not start server
postgres [###]: Examine the log output.
systemd[l]: vpostgres.service: Control process exited, code=exited, status=1/FAILURE
systemd[l]: vpostgres. service: Failed with result 'exit-code'.
systemd[l]: Failed to start VMware Postgres database server.
  
  
• The /var/vmware/vpostgres/current/pgdata/log/postgres-<date>.log files on the Primary Cell show errors of the form:
  
  <timestamp> [###] LOG: invalid connection type "<string>"
<timestamp> [###] CONTEXT: line <line_number> of configuration file "/var/vmware/vpostgres/current/pgdata/pg_hba.conf"
  
  
• The Primary Cell was recently restarted.

Environment

• VMware Cloud Director 10.5.x
• VMware Cloud Director 10.6.x

Cause

This issue occurs if there are invalid entries in the /var/vmware/vpostgres/current/pgdata/pg_hba.conf file on the Primary Cell.
Cloud Director automatically controls this file and appends all text from any file in the /opt/vmware/appliance/etc/pg_hba.d/ directory (except the existing file /opt/vmware/appliance/etc/pg_hba.d/README).
If invalid entries are present in files that have been created in that directory then they will stop vpostgres from starting.

Resolution

To resolve the issue remove or edit any files that have been added to the /opt/vmware/appliance/etc/pg_hba.d/ directory.
Only valid entries must be present as per the Cloud Director documentation, Configure External Access to Your VMware Cloud Director
Database.

Once the invalid entries or files have been removed wait at least 60 seconds for Cloud Director to update the /var/vmware/vpostgres/current/pgdata/pg_hba.conf and then confirm they are no longer appended to the end of the file.

The vpostgres service can then be restarted:

systemctl restart vpostgres.service