VMware Cloud Director intermittently loses connection to vCenter due to database authentication failure

Products

• VMware Cloud Director

Issue/Introduction

Scenario 1: vCenter server keeps disconnecting

• VCD intermittently loses and re-establishes connection with the vCenter server.

Scenario 2: VM import operations failing

• VM import tasks fail with database authentication errors.

• Manual restarts of the vmware-vcd service hang or fail.

• Consistent database login failure errors in vcloud-container-debug.log:

FATAL: password authentication failed for user "vcloud" org.hibernate.exception.GenericJDBCException: Cannot open connection org.postgresql.util.PSQLException: FATAL: password authentication failed for user "vcloud"

Environment

VMware Cloud Director 10.5.1

Cause

This issue occurs when the PostgreSQL database password is changed at the backend but not properly synchronized across all VCD cells using the required cell-management command.

Each VCD cell maintains its own local DB credentials, and failure to update all cells results in broken authentication.

Resolution

NOTE: Ensure that snapshots of all VCD cells are taken and a backup of the database is created before proceeding.

Follow the steps below to synchronize the updated database password across all VCD cells:

1. Verify DB password consistency on all VCD cells:

   • Check the following files for the dbpassword field:

     grep -i dbpassword /opt/vmware/vcloud-director/etc/global.properties
     grep -i dbpassword /opt/vmware/vcloud-director/etc/responses.properties
     

   • Compare the output from all VCD cells to ensure the passwords are consistent.

   • If any mismatch is found, proceed with the following steps to reconfigure the database connection.

2. Run the reconfigure-database command on each VCD cell:

   /opt/vmware/vcloud-director/bin/cell-management-tool reconfigure-database -dbpassword 'same_password_used_in_ALTER_command' \
   -dbuser vcloud -private-key-path /opt/vmware/vcloud-director/id_rsa --remote-sudo-user=postgres -i $(service vmware-vcd pid cell) 

3. Restart the VCD service on each cell:

   systemctl restart vmware-vcd
   

4. Verify database connectivity via logs:

   tail -f /opt/vmware/vcloud-director/logs/vcloud-container-info.log

     5. Reconnect vCenter Server Instance to VMware Cloud Director

Once the database password is properly synchronized across all VCD cells, vCenter server disconnections and VM import issues will no longer occur.

Additional Information

For more details, see: 

Change the PostgreSQL Database Password of Your VMware Cloud Director