vSAN data protection "unable to authenticate user" after replacing vCenter Machine SSL certificate

Products

VMware vSAN VMware vSAN 6.x VMware vSAN 7.x VMware vSAN 8.x

Issue/Introduction

VMware vSAN Data Protection unable to communicate with the vCenter server with the error.

The vSAN Snapshot Service Appliance loses communication with the vCenter Server because the appliance stores a copy of the old vCenter certificate, which is no longer trusted.

Log Snippet:

The key error message in the /storage/log/snapservice/snap-service.log confirms a TLS (Transport Layer Security) validation error:

"error":"Post \"https://ssvapvcqdc01.ssv.wa.lcl/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

tls: failed to verify certificate: The secure connection handshake failed.
x509: certificate signed by unknown authority: This indicates the certificate presented by the vCenter Server is signed by a Root or Intermediate Certificate Authority (CA) that the Snapshot Service Appliance does not recognize or trust. In this specific case, the appliance is still referencing the old CA chain/certificate stored locally, leading to the failure when it checks the new vCenter certificate.

Environment

VMware Snapshot Service appliance (All Versions)

Cause

The issue is caused by the old vCenter machine ssl certificate stored in the Snapshot Service Appliance machine

Resolution

Manually Update vCenter Certificate

The resolution requires manually updating the vCenter Machine SSL certificate within the Snapshot Service Appliance's trust store and re-authenticating the service.

Step-by-Step Procedure

1. Access Appliance Login to the Snapshot Appliance Service VM via SSH. Establish administrative access to the appliance.

2. Review Old Certificate cat /etc/ssl/certs/snapservice/vc_certs.pem Display the currently stored (old) vCenter certificate/chain in the appliance's trust store.

3. Retrieve New Certificate openssl s_client -connect <VCname>:443Connect to the vCenter server over SSL and retrieve the new, currently active certificate chain presented by vCenter. Copy this output.

4. Compare/Verify Compare the output from Step 2 (Old) and Step 3 (New).Confirm they are different, justifying the certificate replacement.

5. Stop Service systemctl stop snapservice Stop the running snapshot service to prepare for certificate and configuration updates

6. Replace Certificate vi /etc/ssl/certs/snapservice/vc_certs.pem Delete the existing content in this file and paste the new certificate chain copied from Step 3. Save and close the file.

7. Check for the username and password information in the yaml file, if the user name and password information is not provided, edit the file using VI editor vi /opt/vmware/snapservice/app/deployment/compose_snapservice_config.yaml Add or update the SSO username and password details to re-establish the service's authentication token with vCenter.

Sample information is provided below, Update the SSO information of the Vcenter server within the quotes

vsphereConfig:
useMockServer: false
vcenter: <vCenter FQDN>
username: ""
password: ""
sslVerify: true
appliance: <Snapservice Appliance name>
svcAccountCredentialPath: /secrets/svc_account_credential
domainName: vsphere.local

8. Re-Restart Service systemctl start snapservice followed by systemctl start snapservice This final restart applies the new SSO authentication configuration, completing the resolution.

Additional Information

Crucial Note: The certificate in /etc/ssl/certs/snapservice/vc_certs.pem is not automatically updated when the vCenter Machine SSL certificate is replaced, necessitating this manual intervention.