Control Plane Nodes stuck in provisioned state and Cluster certification rotation in stuck state

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• Control Plane Nodes stuck in provisioned state with error "Waiting for a Node with spec.ProviderID vsphere to exist"
• Workload cluster certificate renewal became stuck
• User followed KB 434497 to update thumbprint of vCenter on TCA UI 

Environment

TCA 3.4
TCP 5.0.1

Cause

vCenter thumbprint not getting updated in cpi config map after CaaS VcThumbprint job is successfully finished. This is because vsphere-cpi-addon secret on management cluster contains stale "vsphereconf-custom.lib.txt" item.
This was used for stretched clusters but for obsolete clusters that are not stretched. However the vCenter thumbprint update job did not update the thumbprint recorded in "vsphereconf-custom.lib.txt", nor remove it. This causes the configmap in Workload Cluster is still using the old thumbprint.

Resolution

• Obtain the correct Leaf SHA-1 (Not SHA256) thumbprint using below command:
  
  echo | openssl s_client -connect <vCenter FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha1
  
  
• Edit the CPI secret on the management cluster, stripped the faulty config yaml lines and verify the workload configmap aligned with the correct thumbprint.
  
  
• On the management cluster: remove the "overlay.yaml" and "vsphereconf-custom.lib.txt" lines in the data, and save it.
  
  kubectl get secret -n <cluster_name> <cluster_name>-vsphere-cpi-addon -o yaml 
  
  
• On the workload cluster: check the configmap value, should be same with step1 SHA-1
  
  kubectl get configmap -n kube-system vsphere-cloud-config -o yaml
  
  

Note: This will be fixed on future TCA version