Error "failed to create new client with err: Post "https://<vcenter-fqdn>:443/sdk": host "<vcenter-fqdn>:443" thumbprint does not match "<old-thumbprint>" in workload cluster after replacing vCenter Server certificate

Products

• VMware Telco Cloud Automation

Issue/Introduction

• In the workload cluster some of the vSphere-csi pods the are in a CrashLoopBackOff state.

• Checking the logs reveals connection failures due to a thumbprint mismatch:

  kubectl logs <pod-name> -n <namespace>

"level":"info","time":"2026-03-12T05:06:20.437043751Z","caller":"vsphere/virtualcenter.go","msg":"VirtualCenter.connect() creating new client","TraceId":"####################"}
{"level":"error","time":"2026-03-12T05:06:20.469180061Z","caller":"vsphere/virtualcenter.go:181","msg":"failed to create new client with err: Post \"https://<vCenter FQDN>:443/sdk\": host \"<vCenter FQDN>:443\" thumbprint does not match \"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##\"","TraceId":"####################","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.
  
  
• The TCA Manager, TKG Management Cluster and the TCA control plane and the TKG workload clusters are running in different vCenter Servers.
  
  
• The vCenter Server Certificate on any one or both the vCenter Servers have been replaced.
  
  
• Updating the vCenter Server Certificate to the workload cluster using the update-vc-tp.sh script and recreating the vsphere-csi package installation as per the KB Update vCenter certificate for TKG cluster does not resolve the issue. 

Environment

TCA 3.1.1

TKG 2.5.1

Cause

• This issue occurs when the SSL certificate on the workload vCenter Server is updated, but the new SSL thumbprint is not automatically propagated to the workload clusters.
• The pods in the workload cluster attempt to authenticate using the old thumbprint stored in the Kubernetes vsphere-config-secret, resulting in a mismatch and subsequent connection refusal by the vCenter Server.

Resolution

Manually update the vsphere-config-secret in the workload cluster with the new vCenter Server thumbprint.

1. Identify the new vCenter Server thumbprint Run the following command against your workload vCenter Server to retrieve the current, active SSL thumbprint:
   
   openssl s_client -connect <vcenter-fqdn>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -sha1 | cut -d'=' -f2
   
   Note the output of this command, as you will need it for Step 4 (referred to as <new-thumbprint>).
   
   
2. Connect to the workload cluster Set your kubectl context to the affected workload cluster:
   
   kubectl config use-context <workload-cluster-name>-admin@<workload-cluster-name>
   
   
3. Export the current CSI configuration Extract the current configuration containing the old certificate thumbprint to a local file:
   
   kubectl get secret vsphere-config-secret -n vmware-system-csi -o jsonpath='{.data.csi-vsphere\.conf}' | base64 -d > csi-final.conf
   
   
4. Update the thumbprint in the configuration file Use sed to replace the <old-thumbprint> with the <new-thumbprint> you retrieved in Step 1. (Example: sed -i 's/OLD_THUMBPRINT/NEW_THUMBPRINT/g' csi-final.conf)
   
   sed -i 's/<old-thumbprint>/<new-thumbprint>/g' csi-final.conf
   
   
5. Verify that the file has been updated correctly. Ensure that all thumbprint lines now reflect the new value:
   
   cat csi-final.conf
   
   
6. Apply the updated configuration to the Secret Inject the corrected configuration file back into the Kubernetes Secret:
   
   kubectl create secret generic vsphere-config-secret -n vmware-system-csi --from-file=csi-vsphere.conf=csi-final.conf --dry-run=client -o yaml | kubectl apply -f -
   
   
7.  Restart the CSI Pods Delete the existing CSI controller pods to force them to restart and mount the newly updated secret:
   
   kubectl delete pod -n vmware-system-csi -l app=vsphere-csi-controller