vSphere Kubernetes Cluster Node stuck in Provisioned state - could not validate the identity of the API server: could not find a JWS signature in the cluster-info Configmap for token ID

Products

• VMware vSphere 7.0 with Tanzu
• vSphere with Tanzu

Issue/Introduction

• In a vSphere Kubernetes cluster, one or more nodes are stuck in Provisioned state.
  
  
• While connected to the Supervisor cluster context, the impacted node's machine object shows Provisioned state.
  
  kubectl get machines -n <cluster namespace>
  
  
• The impacted node's VM is powered on and has an IP address assigned.
  
  kubectl get vm -o wide -n <cluster namespace>


• While connected directly to the impacted node through SSH as breakglass user vmware-system-user the cloud-init-output.log shows error messages similar to the below where "abcdef" is the unique token ID that differs between environments.
  
  error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "abcdef"
The cluster-info ConfigMap does not yet contain a JWS signature for token ID "abcdef", will try again


• While connected to the affected vSphere Kubernetes cluster's context, the above token ID is missing from the below configmap.
  
  kubectl get configmap cluster-info -n kube-public -o yaml


• The above token ID has expired.
  
  kubeadm token list | grep <token ID>

TOKEN                  TTL            EXPIRES                USAGES                   DESCRIPTION                                                    EXTRA GROUPS
abcdef.1234567890    <invalid>       YYYY-MM-DDThh:mm:ssZ    authentication,signing  token generated by cluster-api-bootstrap-provider-kubeadm   system:bootstrappers:kubeadm:default-node-token
  
  

Environment

vSphere 7.0 with Tanzu

vSphere 8.0 with Tanzu

This issue can occur regardless of whether or not this cluster is managed by TMC.

Cause

The affected vSphere Kubernetes cluster's certificates expired and affected system services are still failing, referencing the expired certificates.

This issue can still occur if the certificates were renewed, but steps were missed in restarting the necessary services to pick up the certificate change.

Note: This issue can also occur if there is a time synchronization issue between the existing and the newly deployed Control Plane node.

Resolution

The vSphere Kubernetes cluster certificates will need to be renewed and the appropriate services restarted accordingly.

Use the certmgr script included in the following KB article to renew the affected cluster's certificates.

• Replace vSphere with Tanzu Guest Cluster Certificates

If the above certmgr script does not successfully renew the certificates, open a ticket to VMware by Broadcom Support referencing this KB article for assistance in manually renewing the certificates in the affected cluster.

The services may need to be restarted on all control plane nodes in the vSphere Kubernetes cluster but should be restarted on one control plane at a time to ensure that the services are not all down at the same time.