vSphere with Tanzu's ESXi worker nodes stuck in Not Ready state after upgrade to version 8.0

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
Known issue with spherelet build 19554634 and earlier not working with ESXi 8.0. Please see chart below for a spherelet versioning guide. Remember that spherelet updates are done when the supervisor cluster is updated. 

 

vCenter Version	Supervisor Version	Spherelet Version	Issue Happens
vCenter Server 7.0 Update 3f  (Build 20051473)	1.22	19867648	No
vCenter Server 7.0 Update 3e  (Build 19717403)	1.22	19554634	YES
vCenter Server 7.0 Update 3d (Build 19480866)  or earlier	1.21 or earlier	19534025	YES

Spherelet version can be gathered by ssh-ing into the host and running "esxcli software vib get -n spherelet"

[root@esx-01a:~] esxcli software vib get -n spherelet
VMware_bootbank_spherelet_1.3.0-18485732
   Name: spherelet
   Version: 1.3.0-18485732

Issue happens when updating from certain version of vCenter and Supervisor cluster to vSphere 8.0, THEN updating ESXi before the Supervisor cluster. Recommended upgrade path is vCenter -> Supervisor Cluster -> ESXi hosts. This issue happens when upgrade path is vCenter -> ESXi without updating the supervisor cluster.

Symptoms: 
1. vSpherePods(PodVMs) cannot be scheduled on these nodes.
2. All hosts will show as NotReady when running kubectl get nodes

Example:
root@422a9a12b8b58e9c218576d6162a9d68 [ ~ ]# kubectl get no
NAME STATUS ROLES AGE VERSION
422a650a7e43e87a71cd85525107f22d Ready control-plane,master 7d18h v1.21.0+vmware.wcp.2
422a7faa7f1dbce0da5336da6613cb8c Ready control-plane,master 7d18h v1.21.0+vmware.wcp.2
422a9a12b8b58e9c218576d6162a9d68 Ready control-plane,master 7d18h v1.21.0+vmware.wcp.2
host01.vmware.com NotReady agent 7d18h v1.21.0-sph-6471c90
host02.vmware.com NotReady agent 7d18h v1.21.0-sph-6471c90
host03.vmware.com NotReady agent 7d18h v1.21.0-sph-6471c90

Environment

VMware vCenter Server 7.0.3

Cause

Certain Spherelet versions from 7.X VC is not compatible with 8.0 ESXi hosts

Resolution

Perform WCP Cluster upgrade to the affected supervisor cluster (since vCenter is on 8.0, there will be a supervisor upgrade available).

Workaround:
There is no workaround to this issue.

Additional Information

Impact/Risks:
WCP Cluster will not be able to schedule any workloads, since the communication with hosts is stranded due to spherelet crashing, which results from spherelet version mismatches with ESXi version.