VKS Guest Cluster nodes stuck in "Provisioned" state with message "NodeHealthy: Waiting for a Node with spec.providerID"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• VKS Guest cluster node post re-conciliation/rollout gets stuck in "Provisioned" state.
  
  
• The affected node can be both Control Plane and Worker node.
  
  
• CAPI is unable to transition the status of the node to a "Running" state because it is waiting for the Guest Cluster Cloud Provider Interface to append "providerID" under the node specification. Below is what is seen in the describe output of the concerned machine.
  
  Message: * NodeHealthy: Waiting for a Node with spec.providerID vsphere://<ID> to exist
* Control plane components: Waiting for a Node with spec.providerID vsphere://<ID> to exist
* EtcdMemberHealthy: Waiting for a Node with spec.providerID vsphere://<ID> to exist
  
  
• The affected node is unreachable to the api server load balancer IP on port 6443. To validate the same, run the following command :
  
  curl -v -k <IP of the api svc lb>:6443
  
  
• Migrating the Virtual Machine to another ESXi host fixes the issue. 

Environment

• VMware vSphere Kubernetes Service
• VMware Cloud Foundation

Cause

This can be caused due to an underlying network issue with the ESXi host where the affected node Virtual Machine resides. 

Resolution

Validate and remediate the underlying ESXi host network issues.