How to resolve "Failed to remove Component VMware-Spherelet" error during ESXi remediation

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During ESXi host image remediation—typically performed after disabling VMware vSphere with Kubernetes (Supervisor Cluster)—the task fails on specific hosts with the error: "Failed to remove Component VMware-Spherelet, files may still be in use."

Additionally, checking the Spherelet service on the affected host may reveal service initialization failures, such as BuildK8sClientConfig failed, often linked to corrupted (0-byte) configuration files in /etc/vmware/spherelet/.

Environment

VKS 8.x,9.x

Cause

The Spherelet service is in a hung or locked state, preventing the VIB uninstallation task from completing successfully. This is often caused by:

• Corrupted configuration files (0-byte size) within the Spherelet directory.
• A stale WCP (Workload Control Plane) registration that prevents the vCenter remediation task from correctly stopping the service.

Resolution

This issue can be resolved by manually forcing the node removal and VIB uninstallation (often referred to as the "sledgehammer method").

Prerequisites:

• Ensure the ESXi host is in Maintenance Mode.
• Verify you have SSH access to both the Supervisor Control Plane VM and the affected ESXi host.

Steps:

1. Remove the stale node object: SSH into one of your Supervisor Control Plane VMs and execute: kubectl delete node <node-hostname>

2. Manually remove the Spherelet VIB: SSH into the affected ESXi host and remove the VIB: localcli software vib remove -n spherelet

3. Trigger WCP Reconfiguration: Exit Maintenance Mode on the ESXi host via vCenter. This will force the Workload Control Plane to detect that the host requires reconfiguration and redeploy the necessary Spherelet components cleanly.

4. Verify: Once the host exits Maintenance Mode, the Spherelet service should initialize successfully, and the host status should return to "Ready".