1 of 2 downstream nodes not attached

Products

• VMware Cloud Director

Issue/Introduction

• Checking the repmgr node status you see "1 of 2 downstream nodes not attached". 
• Checking repmgr node status on the specified node may show a high value for Replication lag.
• Checking repmgr cluster show may show all nodes are running without errors. 

Environment

• 9.x
• 10.x

Cause

This issue can occur if the standby cell is not able to replicate the database from the primary. 

Resolution

!!! GRACEFULLY SHUTDOWN ALL CELLS AND TAKE SNAPSHOTS PRIOR TO MAKING ANY DATABASE CHANGES !!!
!!! IF YOU DO NOT FEEL COMFORTABLE PERFORMING THESE STEPS PLEASE CONTACT BROADCOM SUPPORT FOR ASSISTANCE !!!

Clone the database from the Primary Cell to the Standby Cell. 

1. Remove the Standby from the Repmgr Cluster. 

1. Obtain the ID of the failed cell.
   # sudo -i -u postgres repmgr cluster show
   
   
2. Un-register the failed cell from the Repmgr Cluster. 
   # sudo -i -u postgres repmgr standby unregister -f  /opt/vmware/vpostgres/current/etc/repmgr.conf --node-id=xxxxx

2. Delete the Postgres Database from the Failed Standby Cell. 

1. Stop the vCD Services.
   # service vmware-vcd stop
2. Stop the Postgres Service.
   # systemctl stop vpostgres.service
   
   
3. Delete the Postgres Database from the failed standby cell.  
   # rm -rf /var/vmware/vpostgres/current/pgdata

3. Copy the Primary Cell Database to the Failed Standby Cell. 

# sudo -i -u postgres repmgr -h <primary_database_ip> -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby clone

4. Start Postgres Service. 

# systemctl start vpostgres.service

5. Register the Standby Cell with the Repmgr Cluster. 

# sudo -i -u postgres repmgr -h <primary_database_ip> -U repmgr -d repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby register --force

6. Verify the Standby Cell is registered to the Repmgr Cluster Successfully and you no longer see the error message or replication lag with the Standby Cell. 

          # sudo -i -u postgres repmgr cluster show

          # sudo -i -u postgres repmgr node status

7. Start the vCD Services. 

# service vmware-vcd start