Service runtime cluster goes out of sync with VC due to mismatch in VC Certs/Thumbprint or both

Products

• VCF Operations
• VCF Automation

Issue/Introduction

• This can happen if periodic automation script is not able to update the VCFA service runtime PD with changed VC certs, or any condition causing mismatch
• VCF Operations upgrade, the workflow failed at the "Stage Fleet Lifecycle Plugin in VCF Services Runtime" task due to certificate-related issues
• When performing operations on VCF Automation 9.x (such as scheduled backups or powering off) from Fleet Management, the task fails with the following error: 
  

  Error Code: LCMVMSP10035 Synthetic checker on the application platform failed UI details show a thumbprint mismatch: Post "https://VC-FQDN/sdk": host "VC-FQDN:443" thumbprint does not match "SHA1-THUMBPRINT"

Environment

VCF Automation 9.0 and 9.1,
VCF Identity Broker 9.0 running on vCenter server 8.0.x

Cause

VCFA and VIDB (underlying Kubernetes platform) can use vSphere NDC (Non-Destructive Certificate rotation) endpoint to update its copy of the vCenter certificate and thumbprint periodically when available. However, the vCenter NDC is available from vSphere 9.0 onwards. When the vCenter certificate is rotated on vSphere 8.0, the copy maintained by VCFA/VIDB is not updated automatically.

Resolution

Interactive script to fix and remediate this problem.

Note: SDDC manager VM to host this script or any host which has reachability to the control plane VM

1. SSH into the SDDC Manager.

2. Run the interactive update script:

    bash update-vcenter-ssl-cert.sh

3. Enter your control plane node IP address:

4. Enter the password for vmware-system-user to establish the connection and elevate to root via sudo

5. Enter the fully qualified domain name of the vCenter Server. 

Below is the example output for the script ran :

=== Step 0: Connect to control plane node ===
Control plane node IP/hostname: <Provide the IP of CONTROL_PLANE_IP>
Password for vmware-system-user:
Connecting to ...
Connected to as vmware-system-user.
Verifying sudo elevation...
Elevated to root on 

=== Preflight ===
Target: pd/vmsp-platform -n vmsp-platform (via )

=== Step 1: vCenter details ===
VCenter FQDN (VC-Host): <Provide vCenter_fqdn>

Fetch the SSL certificate from :443 and apply it to pd/vmsp-platform, secret/platform-trust, and restart vsphere-csi-controller on ? [y/N]: y
=== Step 2: Updating vCenter SSL certificate on ===
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[2].matchExpressions[0].key: node-role.kubernetes.io/master is use "node-role.kubernetes.io/control-plane" instead
Warning: spec.template.spec.containers[4].ports[0]: duplicate port name "prometheus" with spec.template.spec.containers[2].ports[1], services and probes that select ports by name will use spec.template.spec.containers[2].ports[1]
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched (no change)
secret/platform-trust patched (no change)
deployment.apps/vsphere-csi-controller restarted
packagedeployment.releases.vmsp.vmware.com/vmsp-platform condition met

New vCenter thumbprint: <THUMBPRINT_REDACTED>
=== Done ===
Success: pd/vmsp-platform -n vmsp-platform is Ready with the updated vCenter SSL certificate.