Supervisor Host status stuck in "Configuring" because it is unable to drain one of the ESXi Hosts

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The vSphere Supervisor cluster status is stuck in Configuring.
  
  
• One or more ESXi hosts are stuck in a Ready, SchedulingDisabled state while attempting to enter maintenance mode.
  
  
• A Pod (e.g., cci-ns-controller-manager) remains in a Terminating state for an extended period (e.g., several days).
  
  
• Manual attempts to drain the node fail or time out.
  
  
• The Spherelet service lost connectivity to the API server when the Control Plane VM (CPVM) was down. The below entries in /var/run/log/spherelet.log of the ESXi Host confirm the same.
  
  Er(3) spherelet[]: msg="Failed to retrieve node" error="Get \"https://<API_SERVER_IP>:6443/api/v1/nodes/<ESXi_HOSTNAME>\": dial tcp <API_SERVER_IP>:6443: connect: connection refused"
Wa(4) spherelet[]: msg="GetStatus failed" error="failed to query pod status: connection error: desc = \"transport: Error while dialing: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock <VSOCK_ID>:1143\"" namespace=svc-cci-ns-cxq5n pod=cci-ns-controller-manager-<ID>-pkpbr uid=<POD_UID>
Er(3) spherelet[]: msg="error while updating pod status in kubernetes: Patch \"https://<API_SERVER_IP>:6443/api/v1/namespaces/svc-cci-ns-cxq5n/pods/cci-ns-controller-manager-<ID>-pkpbr/status\": dial tcp <API_SERVER_IP>:6443: connect: connection refused" namespace=svc-cci-ns-cxq5n pod=cci-ns-controller-manager-<ID>-pkpbr reason=AgentUnreachable status=Failed
Er(3) spherelet[]: msg="Failed to get pod" error="Get \"https://<API_SERVER_IP>:6443/api/v1/namespaces/svc-cci-ns-cxq5n/pods/cci-ns-controller-manager-b7f64f58d-pkpbr\": dial tcp <API_SERVER_IP>:6443: connect: connection refused" namespace=svc-cci-ns-cxq5n pod=cci-ns-controller-manager-<ID>-pkpbr uid=<POD_UID>

Environment

VMware vSphere Kubernetes Service
VMware vCenter Server

Cause

This issue typically occurs when a PodVM is powered off manually (or via an external action) but not deleted from the inventory while the host is preparing for maintenance mode. When the host attempted to enter maintenance mode, Kubernetes (K8s) tried to drain the node. However, because the PodVM still existed in a powered-off state and the host was cordoned, the termination process could not complete correctly.

Resolution

To resolve this state and allow the host to complete its transition into maintenance mode, perform the following steps:

1. Log in to the Supervisor Cluster and identify pods that are not in the Running state:
   
   kubectl get pods -A | grep -v Running


2. Verify the PodVM State. Check the vSphere inventory for the Virtual Machine corresponding to the stuck Pod. If the VM is powered off, attempt to delete it from the inventory. If the VM cannot be deleted via the UI, you may need to force-delete the Pod at the Kubernetes level using the command below.
   
   kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0
   
   
3. If the host remains stuck after the Pod is removed, a restart of the vCenter Server services or the Management agents on the affected ESXi host may be necessary to clear the stale task.