Cloud Director UI unavailable after replacing the Cloud Director Appliance Management Certificates

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• Cloud Director UI could not be reached after following the Cloud Director documentation to Replace the Appliance Management Key-Certificate Pair.
• The new Appliance Management certificate is signed by an internal certificate authority which would not be recognised by default.
• Cloud Director /opt/vmware/vcloud-director/logs/vcloud-container-debug.log files show an error connecting to the database similar to:

| ERROR    | processor-ContentLibrary  | JDBCExceptionReporter          | SSL error: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection |   

• PostgreSQL /var/vmware/vpostgres/current/pgdata/log/postgresql-<DATE>.log files on the Primary Cell show client connection errors of the form:

LOG:  could not accept SSL connection: sslv3 alert certificate unknown

• Appliance Sync service /opt/vmware/var/log/vcd/appliance-sync.log files on all Cells show that their truststores have been updated with the new Appliance Management certificate:

| Updating VCD trust store - processing directory node-<NODE_UUID>

• The /opt/vmware/vcloud-director/etc/truststore.pem on all Cells contains the new Appliance Management certificate.

Environment

VMware Cloud Director 10.4.1
VMware Cloud Director 10.4.2

Cause

This issue occurs if the Appliance Management certificate is signed by an internal certificate authority which would not be recognised by default.
The Cloud Director service will only trust the certificate if it or the certificate authority certificate is present in its truststore during service startup.

Resolution

For Cloud Director 10.5.1 and later follow the updated steps which include stopping and restarting the Cloud Director services as part of the process, Replace or Renew the VMware Cloud Director Appliance Management Certificates.

To resolve the issue in earlier versions of Cloud Director simply stop the Cloud Director services on the Cells before changing the Appliance Management certificate and start the Cloud Director services on the Cells after the new certificate has been applied.

Example steps would be as follows:

1. Before changing the certificate schedule a downtime and stop the Cloud Director service on all Cells in the cluster, the guest OS of the Cells does not need to be shutdown:
       
       /opt/vmware/vcloud-director/bin/cell-management-tool -u <VCD_ADMIN_USERNAME> cell --shutdown
       
       systemctl stop vmware-vcd
      
2. Proceed to apply the new certificate to the Cells as per the Cloud Director documentation, Replace the Appliance Management Key-Certificate Pair.
3. After replacing the certificate and key, and restarting the Appliance VAMI and PostgreSQL services wait 2 minutes to ensure the Appliance Sync service of all the Cloud Director Cells is able to update the truststores with the Cell's new cert. The Appliance Sync logs can be followed to confirm that this is occurring, it should update approximately every ~60 seconds:
       
       tail -f /opt/vmware/var/log/vcd/appliance-sync.log | grep "Executing vcd appliance sync scripts\|Updating VCD trust store\|Successfully completed run of appliance sync script"
      
4. Confirm the new certificate is present in all the Cells' truststore:
   
       less -i /opt/vmware/vcloud-director/etc/truststore.pem
      
5. Restart the Cloud Director service again on all the Cells:
   
       systemctl start vmware-vcd
      
6. Confirm that the Cloud Director Provider and Tenant UIs become available once the services have finished startup.

Additional Information

Impact/Risks:

Backup Cloud Director before making any changes, Backup and Restore of VMware Cloud Director Appliance.

Stopping the Cloud Director services on the Cells will make Cloud Director unavailable so schedule maintenance for the change as appropriate.