Issue with kubernetes VM Creation in the vCenter

Products

• Tanzu Kubernetes Runtime
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:

•The VM is not created in vCenter.
•The corresponding Machine resource is stuck in the Pending state.

Environment

TCA 2.x, 3.x

Resolution

• If the VM is not created in vCenter and the Machine is in a Pending state, please follow the steps below before restarting the components:

  1. Resource Cleanup
     • Delete the following Kubernetes resources related to the failed provisioning:
       • Use the following command to list machines 
         

         kubectl get machines -A | grep cluster_name | grep Pending

         Use the following command to delete machines
         

         kubectl delete machine <machine-name> -n <namespace>

         This shows all Machines and vSpheremachines that are using a VSphere provider and list their associated names

         kubectl get machines -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.infrastructureRef.name}{"\t"}{.spec.providerID}{"\n"}{end}' | grep vsphere://

         Use this command to delete vSpheremachine and vspherevm with associated provider ID you wish to delete:
         

         kubectl delete vspheremachine VSPHEREMACHINENAME -n NAMESPACE
         kubectl delete vspherevm -n <namespace> <vspherevm>

  2. If the VM is not created in vCenter and the Machine is in a Pending state, please follow the steps below before restarting the components:

     Restart the following pods in the management cluster:
         •    Use the following command to list capv and capi pods 

     kubectl get pods -n capi-system && kubectl get pods -n capv-system

          •    Use the following command to restart capv and capi pods:

     kubectl delete pods --all -n capi-system && kubectl delete pods --all -n capv-system

        •    Use the following command to list nodeconfig-controller

     kubectl get pods -n tca-system | grep nodeconfig-operator

         •    Use the following command to restart 

     kubectl delete pod -n tca-system $(kubectl get pods -n tca-system --no-headers -o custom-columns=":metadata.name" | grep ^nodeconfig-operator-)

      

  3. Validation:
         •    After restarting the components, verify if the objects (VMs) are being created successfully in vCenter.
         •    Confirm that the provisioning completes as expected and nodes are joining the cluster properly.