How to remove a failed Standby Cell

Products

• VMware Cloud Director

Issue/Introduction

• A Standby Cell needs to be removed from the VMware Cloud Director High Availability Cluster. 

Environment

10.x

Resolution

Steps to Remove a Working/Failed/In-Active Standby Cell from Cluster:

Method 1: Remove via GUI

Refer Unregister a Failed VMware Cloud Director Appliance Primary or Standby Cell in a Database High Availability Cluster to remove via UI

Method 2: Remove via CLI

1. Login to primary cell as root user via ssh. 
   
   
2. Run below command to fetch cluster details:
   

   sudo -i -u postgres /opt/vmware/vpostgres/current/bin/repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf cluster show

3. You will see output similar to below one:
   

   ID | Name | Role | Status | Upstream 
    1 | pg-primary | primary | * running | | default | host=#.#.#.# dbname=repmgr user=repmgr
    2 | pg-standby-1 | standby | * running | pg-primary | default | host=#.#.#.# dbname=repmgr user=repmgr
    3 | pg-standby-2 | standby | * running | pg-primary | default | host=#.#.#.# dbname=repmgr user=repmgr

4. Make a note of the failed cell ID number, for example "3"
   
   
5. Run below command by updating with the Cell ID
   

   sudo -i -u postgres /opt/vmware/vpostgres/current/bin/repmgr -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby unregister --node-id=3

    

6. You will see a output similar to below one:
   

   node 3 unregistered successfully

    

7. Power off the Standby VM that was removed from the cluster. 
   
   
   
8. Cleanup the 'node-uuid' folder on the Transfer Directory. 
   
   
   1. Change directory to the 'appliance-nodes' folder on the Transfer Directory. 
      

      cd /opt/vmware/vcloud-director/data/transfer/appliance-nodes

   2. List the contents of the 'appliance-nodes' directory. 
      

      ls -lahtr

   3. The appliance-nodes directory will have a 'node-uuid' folder for each cell. You can determine the folder that relates to each cell by using the cat command on the repmgr-node-name. Find the uuid that corresponds to the removed Standby Cell. 
      

      cat node-uuid/repmgr-node-name

   4. Once you have determined which 'node-uuid' corresponds to the Standby Cell removed from the cluster, we can remove it. 
      

      rm -Rf node-uuid

      Note: This command will forcibly remove a directory and all of its contents recursively without prompting for confirmation. 
      
      
   5. List the contents to ensure the folder has been removed successfully. 

      ls -lahtr

9. Cleanup the 'cells' folder on the Transfer Directory. 
   
   
   1. Change directory to the 'cells' folder on the Transfer Directory. 
      

      cd /opt/vmware/vcloud-director/data/transfer/cells

   2. List the contents of the 'cells' directory. 
      

      ls -lahtr

   3. The cells folder on the Transfer Directory contains uuid's that correspond to each cell uuid registered in the Postgres Database. Use the following command to determine which uuid corresponds to each cell. 
      

      sudo -i -u postgres psql vcloud -x -c "select name,uuid from cells;"

   4. Once you have determined which uuid corresponds to the Standby Cell removed from the cluster, we can the 'uuid' folder from the cells directory in /opt/vmware/vcloud-director/data/transfer/cells/.
      

      rm -Rf 'uuid'

   5. ls -lahtrList the contents of the 'cells' directory to ensure the uuid folder has been removed successfully. 

10. Cleanup the "synchronous_standby_names" in the 'postgresql.conf' file. 
    
    
    1. Change directory to /var/vmware/vpostgres/current/pgdata
       

       cd /var/vmware/vpostgres/current/pgdata/

    2. List the contents of the 'pgdata' directory. 
       

       ls -lahtr

    3. Use cat to show the contents of the postgresql.conf file and grep for 'synchronous_standby_names'. 
       

       cat postgresql.conf |grep 'synchronous_standby_names'

    4. If you DO NOT see the Standby Cell listed in 'synchronous_standby_names' then nothing needs to be done. If you DO see the Standby Cell listed in 'synchronous_standby_names' continue with Steps 5 and 6. 
       
       
    5. Backup the postgresql.conf file prior to editing. 
       

       cp postgresql.conf /var/vmware/vpostgres/current/pgdata/postgresql.conf.bak

    6. Edit the postgresql.conf file using VI. 
       

       vi postgresql.conf

       1. Press Insert on your keyboard to begin editing. 
       2. Navigate to the 'synchronous_standby_names' and delete the Standby cell using the backspace key on your keyboard. 
       3. Press ESC on your keyboard to exit VI. 
       4. Press the colon key which will enter command-line mode. 
       5. Type the following and press Enter on your keyboard to apply the changes. 
          wq!
       6. Use cat to show the contents of the postgresql.conf file and verify your changes were successful.