Grey screen when connecting to vCloud Admin and Provider portal pages.
Connecting to VAMI works, but displays "Cluster Health: Read-Only Primary" on at least one of the nodes.
vCloud Director 10.6.x
This is caused by the Database copies held by the different nodes becoming out of sync, often due to a snapshot reversion that didn't include all nodes.
1) Validate the vCloud Director cluster health by running the following command via ssh on the Primary Cell:
su - postgres repmgr cluster show
If it's degraded, proceed with the following steps.
2) Power down all three cells and take snapshots of them
Use the following process to properly shut down the cells:
https://knowledge.broadcom.com/external/article/308791/steps-to-stop-start-or-restart-the-vmwar.html
3) Power the cells back up then use the following command to turn the vcd service off on all three.
systemctl stop vmware-vcd.service
4) Use the ifconfig command to get the eth1 IP address of the Primary cell.
5) Recover the standby cells by running the following on the STANDBY CELLS ONLY
– stop vpostgres service:
systemctl stop vpostgres
– Remove the current pgdata:
rm -rf /var/vmware/vpostgres/current/pgdata
6) Clone the Database using the Primary IP address (example 192.168.200.1) as follows:
su - postgres -c "/opt/vmware/vpostgres/current/bin/repmgr -d 'host=192.168.200.1 user=repmgr gssencmode=disable' standby clone"
7) Start the vpostgres service:
systemctl start vpostgres
8) Register the cell to the Cluster. Once again, use the IP from the Primary on the following command:
su - postgres -c "/opt/vmware/vpostgres/current/bin/repmgr -h 192.168.200.1 -U repmgr -d remgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby register --force"
9) Run the same commands (steps 5 - 8) on the second Standby cell.
10) Check the Database Cluster health state:
su - postgres repmgr cluster show