1 of 2 downstream nodes not attached

Products

• VMware Cloud Director

Issue/Introduction

• Checking the repmgr node status you see "1 of 2 downstream nodes not attached". 
• Checking repmgr node status on the specified node may show a high value for Replication lag.
• Checking repmgr cluster show may show all nodes are running without errors. 

Environment

• 9.x
• 10.x

Cause

This issue can occur if the standby cell is not able to replicate the database from the primary. 

Resolution

!!! GRACEFULLY SHUTDOWN ALL CELLS AND TAKE SNAPSHOTS PRIOR TO MAKING ANY DATABASE CHANGES !!!
!!! IF YOU DO NOT FEEL COMFORTABLE PERFORMING THESE STEPS PLEASE CONTACT BROADCOM SUPPORT FOR ASSISTANCE !!!

1. Remove the Standby from the Repmgr Cluster. 

1. SSH to Primary Cell.
2. Obtain the ID of the failed cell.
   # sudo -i -u postgres repmgr cluster show
   
   
3. Un-register the failed cell from the Repmgr Cluster. 
   # sudo -i -u postgres repmgr standby unregister -f  /opt/vmware/vpostgres/current/etc/repmgr.conf --node-id=xxxxx

2. Delete the Postgres Database from the Failed Standby Cell. 

1. SSH to Failed Standby Cell.
2. Stop the vCD Services.
   # service vmware-vcd stop
3. Stop the Postgres Service.
   # systemctl stop vpostgres.service
   
   
4. Delete the Postgres Database from the failed standby cell.  
   # rm -rf /var/vmware/vpostgres/current/pgdata

3. Copy the Primary Cell Database to the Failed Standby Cell. 

1. From SSH on failed standby cell run:

# sudo -i -u postgres repmgr -h <primary_database_ip> -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby clone

4. Start Postgres Service on Standby Cell

# systemctl start vpostgres.service

5. While still on SSH of Standby Cell, Register the Standby Cell with the Repmgr Cluster. 

# sudo -i -u postgres repmgr -h <primary_database_ip> -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby register --force

6. Verify the Standby Cell is registered to the Repmgr Cluster Successfully and you no longer see the error message or replication lag with the Standby Cell. 

          # sudo -i -u postgres repmgr cluster show

          # sudo -i -u postgres repmgr node status

7. Start the vCD Services on the Standby Cell. 

# service vmware-vcd start