Supervisor stuck in deleting state after deactivation due to the error [WaitingFoHostNodesDeleted]
search cancel

Supervisor stuck in deleting state after deactivation due to the error [WaitingFoHostNodesDeleted]

book

Article ID: 450209

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

When deactivating Supervisor you see from vSphere UI that it is in Deleting state loop even thought Supervisor Control Plane VMs are deleted. 

Checking wcpsvc log you see something like this

2026-07-27T14:01:20.548Z DEBUG wcp 11008 [vc@4413] [kubelifecycle/controller.go:453] [opID=[OP_ID_1]] Attempting to sync supervisor [SUPERVISOR_ID_1] state with desired state.
2026-07-27T14:01:20.549Z DEBUG wcp 11008 [vc@4413] [logger/trace.go:77] [opID=[OP_ID_2]] [BEGIN] [kubelifecycle.(*Controller).syncKubeInstanceState:494] supervisor=[SUPERVISOR_ID_1]
2026-07-27T14:01:20.549Z INFO wcp 11008 [vc@4413] [kubelifecycle/spherelet_lifecycle.go:191] [opID=[OP_ID_2]] Unconfigure spherelet from all hosts in supervisor [SUPERVISOR_ID_1]
2026-07-27T14:01:20.549Z DEBUG wcp 11008 [vc@4413] [kubelifecycle/node_controller.go:1212] [opID=[OP_ID_2]] For node [HOST_ID_1], setting desiredState from Disabled to Disabled
desiredState from Disabled to Disabled
2026-07-27T14:01:20.549Z DEBUG wcp 11008 [vc@4413] [kubelifecycle/node_controller.go:1212] [opID=[OP_ID_2]] For node [HOST_ID_11], setting desiredState from Disabled to Disabled
2026-07-27T14:01:20.549Z DEBUG wcp 11008 [vc@4413] [kubelifecycle/controller.go:2628] [opID=[OP_ID_2]] Wait for hosts to be cleanup from the cluster.
2026-07-27T14:01:20.549Z WARNING wcp 11008 [vc@4413] [kubelifecycle/controller.go:536] [opID=[OP_ID_2]] Unable to disable supervisor [SUPERVISOR_ID_1] because of the reason [WaitingFoHostNodesDeleted]. Err <nil>

Environment

VCF 9.1 

Cause

NSX cleanup was initiated but never got a success nor failure generated following this call.

Resolution

Engage Broadcom Support to clean up the stale Supervisor from DB.