Data Services Manager connection to VCF Automation fails with "failed to verify certificate" following VCF Automation certificate update.

Products

• VCF Automation

Issue/Introduction

Following the installation of a new certificate on the VCF Automation (VCFA) VIP, the Data Services Manager (DSM) fails to establish a connection to VCFA. The DSM connection status remains in a 'Pending' state, and trace identifiers indicate that the TLS handshake fails at the verification stage with following error in the UI.

DSM vcfabinding is not ready: failed to create VCFA TM client: Get "https://<dns>l:443/api/versions": tls: failed to verify certificate: x509: certificate signed by unknown authority.

• Validate ingress-vcfa-tls-external secret tls.crt missing the full certificate chain:

kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

Environment

VCF Automation 9.x

Cause

VCFA fails to propagate the full certificate chain to the Supervisor cluster and Operations component. This prevents downstream services, such as DSM, from establishing secure TLS communication with VCFA services.
VCF Operations controls the certificate chain lifecycle and pushes it to a designated location for the Service Manager to inject across all VCF Services. In this instance, an incomplete or invalid chain at the source propagates to the DSM appliance, breaking the required trust anchor for TLS orchestration.

Resolution

To manually force the propagation of the full certificate chain (including intermediate and root CAs) into the OPS component inventory and supervisor cluster to correctly establish the trust anchor, perform the following steps:

1. If not available, collect and compose the full chain of the certificate in one pem file.
   • Depending on the certificate used, this could be Leaf, Intermediate and Root certificates in one file.
     
     
2. Encode the full certificate chain:
   

   cat <full certificate>.pem | base64 -w 0

    

3. SSH into the VCFA appliance VM as vmware-system-user
   
   
   
4. Establish root and Kubernetes access:
   

   sudo su
   export KUBECONFIG=/etc/kubernetes/admin.conf

    

5. Confirm that the ingress-vcfa-tls-external secret's tls.crt is missing the full certificate chain:
   

   kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d
   
   kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

    

6. Take a backup of the ingress-vcfa-tls-external secret:
   

   kubectl get secret -n prelude ingress-vcfa-tls-external -o yaml > ingress-vcfa-tls-external-backup.yaml

    

7. Edit the ingress-vcfa-tls-external secret and change tls.crt to the full certificate chain that was encoded in Step 2:
   

   kubectl edit secret -n prelude ingress-vcfa-tls-external

    

8. Confirm that the secret was updated with the full certificate chain:
   

   kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d

    

9. Initiate a sync and immediate reconciliation, pushing the updated certificate chain to the Supervisor cluster:
   

   for name in $(kubectl get supervisorregistries.vcd.addon.vcf.vmware.com -n prelude -o name); do
     kubectl annotate -n prelude "$name" \
       reconcile-trigger="$(date +%s)" --overwrite
   done

   This command updates the timestamp of this supervisorregistries custom resource to the current date.

   It can take up to 10 minutes to propagate to the Supervisor cluster.