ESXi Host Stuck Exiting/Entering Maintenance Mode at 95% due to sub-task, 'Check Namespaces node ready operation', gets stuck at 65% while Configuring Spherelet Vib

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• ESXi Host fails to complete the process of entering or exiting Maintenance Mode, stalling consistently at 95%. The underlying sub-task, specifically the 'Check Namespaces node ready operation', gets stuck at 65% during the 'Configuring Spherelet Vib' phase.
  
  
• When checking the Config Status of the Supervisor under the Workload Management section of the vSphere Client, the following specific error message is observed on all Supervisor control plane nodes:
  
  System error occurred on Master node with identifier ################################. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', 'fluentbit-config-system', '--namespace', 'vmware-system-logging', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1..

Environment

• vSphere Kubernetes Service

Cause

• vSphere Supervisor Root Disk Space Full at 100%: All three vSphere Supervisor Control Plane Node VMs have their root disk partition completely utilized, showing 100% usage. The lack of disk space prevents the successful execution of critical Kubernetes and logging commands (like the kubectl command shown in the error details for log forwarding configuration), which are essential for the Spherelet VIB configuration and overall Supervisor health check when an ESXi host's status changes.

Resolution

• Follow the steps detailed in the Broadcom Knowledge Base article 'vSphere Supervisor Root Disk Space Full at 100%' to safely identify and clear unnecessary files from the vSphere Supervisor Control Plane Node root partition. Once the root disk space has been successfully cleared and falls below the critical threshold (ideally <80%), the ESXi host should automatically or manually be able to complete the stuck Maintenance Mode task (either entering or exiting), and the Supervisor Config Status should return to a healthy state.