Control plane stuck in provisioned state

Products

• VMware Telco Cloud Automation

Issue/Introduction

One of the  control plane machine in the management clusters is stuck in provisioned state.

On running kubectl get node -o wide , there is no external ip assigned.

Verified the  the vsphere-cloud-controller-manager pod logs and it complains "VCenter thumbprint does not match"

Environment

TCA 3.2

Cause

This is due to the VCenter thumbprint does not not match.
The thumprint were updated manually and not via script due to which it was missed to be updated in the Management cluster. To resolve this, thumbprint needs to be updated manually.

 

Resolution

 

1. Verify the thumbprint of the vcenter by obtaining thumbprint from the Obtain vSphere Certificate Thumbprints page and compare it with the vcenter thumbprint in config map
   
   
2. Update the vcenter thumbprint in config map kube-system/vsphere-cloud-config
   
   Login to the Management cluster
   Edit the config map:
   
   kubectl edit configmap -n kube-system sphere-cloud-config -o yaml         //replace the thumbprint with the exact VC thumbprint
   
   
3. Restart by deleting  all 3 cloud pods:
   
   kubectl get pod -A |grep cloud
kubectl delete pod sphere-cloud-controller-manager-<abcde>
kubectl delete pod sphere-cloud-controller-manager-<fghi>
kubectl delete pod sphere-cloud-controller-manager-<jklm>