Unable to update information in the database in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• When attempting to perform an update to the postgres database on the primary VMware Cloud Director appliance, the command hangs without completing and the following is displayed when the call is aborted:
  
  The transaction has already committed locally, but might not have been replicated to the standby.
  
  
• The standby cells were recently removed so that only the primary database node remains.
• In the appliance management interface (VAMI), the cluster only shows a single primary database and the cluster is listed as healthy.
• Running the following command from the cell running the primary database only shows a single running primary node:
  
  repmgr cluster show

Environment

VMware Cloud Director 10.6.x

Cause

Even though there was no longer a postgres replication cluster, there is still information for one or more of the standby cells in the following transfer directory location:

/opt/vmware/vcloud-director/data/transfer/appliance-nodes

Periodically, this information will be reviewed and the postgres configuration will be updated automatically with the standby cell details. If data cannot be replicated to the configured standby cell(s), the errors noted will result.

Resolution

As a precaution, it is recommended to back up your VMware Cloud Director implementation before performing any changes to the configuration, as detailed in the following document:

Backup and Restore of Your VMware Cloud Director Appliance

Ensure that the deprecated standby cells are either powered off or no longer have access to the transfer directory to prevent the appliance node information from from being recreated and then use the VMware Cloud Director API to remove the deprecated standby nodes, as documented in the item below:

Unregister a Running Standby Cell in Your VMware Cloud Director Database High Availability Cluster

If this does not correct the behavior, perform the following additional steps:

1. Navigate to the appropriate directory on the transfer directory from an active VMware Cloud Director cell command line session:
   
   cd /opt/vmware/vcloud-director/data/transfer/appliance-nodes/
   
   
2. Identify the invalid directories by reviewing the repmgr-node-name file in each node directory (node-########-####-####-####-############):
   
   cat node-########-####-####-####-############/repmgr-node-name
   
   
3. Back up the directories to be removed by copying them to another system or location. Alternatively, collect a log bundle from a healthy cell, which will contain a copy of the appliance-nodes directory:
   
   Collecting diagnostic information for VMware Cloud Director
   
   
4. Remove the invalid directories:
   
   rm -r node-########-####-####-####-############