vsphere-csi pods stuck in CrashLoopBackOff due to vCenter thumbprint mismatch in VCF Automation 9.0.x

Products

• VCF Automation

Issue/Introduction

• In VCF Automation 9.0.x environments, the vSphere CSI pods are stuck in a CrashLoopBackOff state.

  kubectl get pods -A | egrep "NAME|csi"
  NAMESPACE      NAME                                READY   STATUS               
  kube-system     vsphere-csi-controller-<ID>        5/7     CrashLoopBackOff   
  kube-system     vsphere-csi-node-<ID>              2/3     CrashLoopBackOff 
  

• kubectl logs deploy/vsphere-csi-controller -n kube-system -c vsphere-csi-controller shows a  a thumbprint mismatch error:

  "level":"error","time":"YYYY-MM-DDDTHH:MM:SS.XXXXXXXZ","caller":"vsphere/virtualcenter.go:690","msg":"failed to connect to VirtualCenter host: \"vcenter.fqdn\". Err: Post \"https://vcenter.fqdn:443/sdk\": host \"vcenter.fqdn:443\" thumbprint does not match 

Environment

VCF Automation 9.0.x

Cause

The SHA1 Thumbprint of the vCenter Server stored in the vsphere-config-secret does not match the current vCenter certificate thumbprint, typically following a certificate renewal or replacement.

Resolution

1.Validate vCenter Thumbprint 

1. Login to VCF Appliance as vmware-system-user, switch user to root :
       sudo su -
2. Retrieve the thumbprint currently stored in the CSI secret:
      kubectl get secrets vsphere-config-secret -n kube-system -o jsonpath='{.data.csi-vsphere\.conf}' | base64 -d
3. Verify the actual vCenter certificate thumbprint:
     echo | openssl s_client -bind $(ip -4 addr show eth0 | grep -m 1 -oP '(?<=inet\s)[0-9.]+') -connect <vCenter-IP>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1

2.Update vCenter Thumbprint

In VCF Automation 9.0.x, vCenter thumbprint updates for vSphere CSI controllers are managed via the vmsp-platform package deployment.Follow the detailed update procedure documented in SSL is out of sync in VCF Automation and Identity Broker 9.0.x running on vCenter 8.0.x .