Supervisor Cluster Deployment Stuck at "Installed and Started Kubernetes Node Agent on the ESXi Host"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The Supervisor Cluster deployment is stuck at the Configuring Host Nodes step.
• From the Workload Management page in vSphere web UI, the deploying Supervisor cluster shows that its Host Config Status is Configuring:
• When viewing details on the Host Config Status that is Configuring, the following task is pending:
  
  Installed and Started Kubernetes Node Agent on the ESXi Host
Kubernetes Node Agent stopped on the ESXi Host for MaintenanceMode
  
  

• There are repeated Apply Solution tasks in the vSphere web UI.
  
  
  
  
• When viewing the wcpsvc logs in the vCenter Server Appliance (VCSA), the following error message is present, where values in <> brackets will vary by environment:
  
  YYYY-MM-DDTHH:MM:SS.sss error wcp [pman/client.go:659] [opID =< opid> -< host-id>-VLCM: AddHost :< cluster-id> :< host-id>] PMan API: Apply Task API: Attempt#[11]: Failed Attempts: [0] of MaxFailedAttemps[1]Error Message: Solution specification in the image are incompatible with hosts '<esxi-hostname-1>, <esxi-hostname-2>, <esxi-hostname-3>'

Environment

• VMware vCenter 8.x
• VMware ESXi 8.x
• VMware vSphere Supervisor
• ESXi Hosts managed by vSphere Lifecycle Manager

Cause

During the initial deployment of a Supervisor Cluster, the spherelet VIB agent is installed on the ESXi hosts which are part of the cluster. If the ESXi hosts are managed by vSphere Lifecycle Manager, the spherelet VIB will be unable to install if the ESXi hosts are considered to be out of compliance or incompatible as per vSphere Lifecycle Manager.
Issue can also happen if the ESXi hosts' boot device is booting with legacy boot option (BIOS) which leads to to the hosts being out of compliance. 

Resolution

Ensure that all ESXi hosts for the Supervisor Cluster are in compliance with vSphere Lifecycle Manager.

While logged into the vSphere web UI:

1. Navigate to the affected Cluster and click on the Updates tab
2. Click on Image under Hosts in the Updates tab's navigation bar
3. Update the corresponding Images as necessary, such as selecting the desired ESXi Version
4. (Optional) If you are using an OEM custom image, select the appropriate Vendor Addon
5. Save Changes
6. Check compliance, resolve the compliance errors as necessary. You can search in Broadcom KB for the failing compliance checks to understand the cause of the issue.
7. If the ESXi host is still showing as non-compliant, remediate the host as necessary
8. Once all ESXi hosts are in compliance, the spherelet VIB will be installed on the ESXi host and Supervisor deployment gets completed

Additional Information

For more information regarding ESXi Host compliance with vSphere Lifecycle Manager, see: Check the Compliance Against a Single Image
For more information about depreciation of legacy boot modes, see: Depreciation of Legacy Boot support