Supervisor Cluster Stuck in "Configuring" State Due to "start-service-ip-repair-controllers failed: unable to perform initial IP and Port allocation check"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The Supervisor Cluster remains stuck in the Configuring state in the vSphere Client UI.
• Core Supervisor services (such as TKG or Velero) fail to initialize or complete reconciliation.
• The spherelet daemon on ESXi hosts reports connection refused or context deadline exceeded when attempting to communicate with the Supervisor Control Plane IP on port 6443.

1. Log in via SSH to the impacted Supervisor Control Plane VM.
2. Run crictl ps -a to identify the restarting kube-apiserver container.
3. Inspect the API server logs:

       crictl logs $(crictl ps -a --name kube-apiserver -q | head -n 1)

       Confirm the presence of the following fatal error at the tail of the log:

       F0810 06:31:12.108798       1 hooks.go:204] PostStartHook "start-service-ip-repair-controllers" failed: unable to perform initial IP and Port allocation check

Environment

VMware vSphere Kubernetes Service

Cause

The kube-apiserver enters a crash loop due to stale etcd locks causing a fatal error in the start-service-ip-repair-controllers post-start hook. This prevents communication on port 6443, holding the Supervisor Cluster in a perpetual Configuring state.

Resolution

A graceful reboot of the impacted Supervisor Control Plane VM flushes stale in-memory database locks, clears open socket states, and allows control plane components (etcd, kubelet, and kube-apiserver) to initialize in the correct sequential order.

Step 1: Perform a Graceful Reboot

Connect to the impacted Supervisor Control Plane VM via SSH and execute a graceful reboot:

reboot

(Alternatively, perform a Guest OS Reboot on the Control Plane VM directly within the vSphere Client UI).

Step 2: Post-Reboot Verification

Allow 3 to 5 minutes for the VM services to complete initialization, then perform the following checks:

         1. Verify Container Status: Ensure all static control plane pods show as Running:

         crictl ps

         2. Verify the nodes: Ensure all the nodes show as Rady:

              kubectl get nodes

         3. Verify vSphere UI: Confirm that the Supervisor Cluster status in the vSphere Client automatically transitions from Configuring to Running.