Error: "1 of 2 downstream nodes not attached" during repmgr status check in VCD

Products

• VMware Cloud Director
• VMware Telco Cloud Platform

Issue/Introduction

• Checking the repmgr node status displays the message "1 of 2 downstream nodes not attached", accompanied by high replication lag on the specified node, while repmgr cluster show indicates all nodes are running.
• This article describes the steps to recover a VMware Cloud Director appliance database cluster when the cluster enters an inconsistent state due to extended network connectivity interruptions.

Environment

• VCD: 10.x
• TCP: 3, 4.x, 4.0.1, 5.x, 5.0. 5.0.1, 5.0.2

Cause

The standby cell is unable to replicate the database from the primary cell, resulting in a broken replication state.

Resolution

1. Gracefully shut down all cells and take a full backup or snapshot of all VCD appliances before making database modifications.

2. Establish an SSH session to the primary cell as root and obtain the node ID of the failed standby cell:

   sudo -i -u postgres repmgr cluster show
   

3. Unregister the failed standby cell from the repmgr cluster:

   sudo -i -u postgres repmgr standby unregister -f /opt/vmware/vpostgres/current/etc/repmgr.conf --node-id=#####
   

4. Establish an SSH session to the failed standby cell, stop the VCD service, and stop the database service:

   service vmware-vcd stop
   systemctl stop vpostgres.service
   

5. Remove the corrupted PostgreSQL database files on the failed standby cell:

   rm -rf /var/vmware/vpostgres/current/pgdata
   

6. Re-clone the database from the primary cell to the failed standby cell:

   sudo -i -u postgres repmgr -h ##### -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby clone
   

7. Start the database service on the standby cell:

   systemctl start vpostgres.service
   

8. Register the standby cell back into the repmgr cluster:

   sudo -i -u postgres repmgr -h ##### -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby register --force
   

9. Verify that replication is healthy and node status errors are resolved:

   sudo -i -u postgres repmgr cluster show
   sudo -i -u postgres repmgr node status
   

10. Start the VCD service on the standby cell:

    service vmware-vcd start
    

 

For complex database synchronization issues, contact Broadcom Support.