ESXi host status "Not Ready" in Supervisor Cluster
search cancel

ESXi host status "Not Ready" in Supervisor Cluster

book

Article ID: 430987

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime VMware vSphere Kubernetes Service VMware vSphere ESXi

Issue/Introduction

  • In a VMware Cloud Foundation (VCF) or vSphere Kubernetes environment, an ESXi host may transition to a Not Ready state. This prevents the Supervisor Cluster from scheduling workloads (vSphere Pods, Kubernetes Grid clusters) on the affected host.
  • The issue can be verified by running the following command in the Supervisor Cluster.

root@<supervisor-id> [ ~ ]# kubectl get nodes

NAME                STATUS           ROLES                 AGE     VERSION

<supervisor-1>   Ready      Control-Plane,master   <days>   <supervisor-version>
<supervisor-2>   Ready      Control-Plane,master   <days>   <supervisor-version>
<supervisor-3>   Ready      Control-Plane,master   <days>   <supervisor-version>
<esxi-host-1>    Ready      agent                  <days>   <spherelet-version>
<esxi-host-2>    Ready      agent                  <days>   <spherelet-version>
<esxi-host-3>    NotReady   agent                  <days>   <spherelet-version>

  • The following command can be run in the affected ESXi host.

root@<esxi-host># /etc/init.d/spherelet status

YYYY-MM-DD hh:mm:ss init.d/spherelet spherelet is not running

Environment

  • vSphere 8.x
  • VMware vSphere Kubernetes Service (VKS)

Cause

The spherelet service, which acts as the kubelet equivalent for ESXi, is not running. This prevents the host from sending health heartbeats to the Supervisor control plane, leading to the "Not Ready" state.

Resolution

Follow these steps to start the service and restore host connectivity in the Supervisor:

  1. Log in to the affected ESXi host as root via SSH.
  2. Check if the spherelet service is active by running the following command: /etc/init.d/spherelet status

  3. If the service is stopped, start it manually: /etc/init.d/spherelet start

  4. Wait 30 seconds and verify the service remains in a "running" state: /etc/init.d/spherelet status

  5. Return to the vSphere Client UI or run the kubectl get nodes command in the supervisor to confirm the host transitions back to Ready state. This typically occurs within 1–2 minutes of service restoration.

 

If the host status transitions to Ready, SchedulingDisabled (or remains SchedulingDisabled after service restoration), the Kubernetes scheduler is explicitly configured to avoid placing new workloads on this host. This state often indicates the node is cordoned or still in Maintenance Mode.

Follow these steps to restore the host to a fully schedulable state:

  1. Check vCenter Maintenance Mode: Ensure the ESXi host is not in Maintenance Mode within the vSphere Client. If it is, exit Maintenance Mode. 

  2. Uncordon the Node: If the host is not in Maintenance Mode, run the following command from your Supervisor Cluster context to uncordon it: kubectl uncordon <ESXi_Host_FQDN>

  3. Verify Status: Run kubectl get nodes in the Supervisor Cluster to confirm the host transitions to a Ready state without the SchedulingDisabled qualifier.

Additional Information

Japanese KB: スーパーバイザクラスタに参加しているESXiホストが"Not Ready"状態になる

Note: The delay in pods reaching Ready (stuck in ContainerCreating) is expected when the underlying Supervisor infrastructure is unstable. Pods cannot be scheduled or have their volumes attached until the Supervisor's "VM Operator" and "CSI" (storage) drivers are fully initialized and have a consistent view of the ESXi inventory