vSphere Supervisor Workload Cluster TKR Incompatible/TKR Compatible False after vCenter Upgrade

Products

VMware vSphere Kubernetes Service VMware vSphere 7.0 with Tanzu vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

After a vCenter upgrade, a vSphere Supervisor Workload Cluster TKC shows as Ready False.

A workload cluster running on an incompatible TKR version cannot be modified or upgraded and will need to be recreated on a compatible TKR version.

While connected to the Supervisor cluster context, the following symptoms are present:

For non-classy clusters, the affected TKC object shows Ready False state:

kubectl get tkc -n <affected cluster namespace>

NAMESPACE        CLUSTER            CONTROL PLANE   WORKER    TKR NAME            AGE         READY
<namespace>     <cluster-name>        #             #       <TKR+version>         #d          False

The TKR used by the TKC in Ready False state also shows as Ready and Compatible False:
```
kubectl get tkr

NAME                 VERSION             READY        Compatible
<TKR---version>    <TKR+version>       False        False
```
- The vCenter Server with TKR Interoperability Matrix can be used to determine Tanzu Kubernetes Releases (TKR) compatibility with vCenter versions.

Environment

VMware vSphere 7.0 with Tanzu

VMware vSphere 8.0 with Tanzu

TKG service supervisor service is not installed.

This issue can occur on workload clusters regardless of whether or not it is managed by Tanzu Mission Control (TMC)

Cause

The environment's vCenter has been upgraded before upgrading all workload clusters to a TKR version that is compatible with the desired vCenter version.

As a result, the workload clusters running on incompatible TKR versions are now stranded, unmanageable and cannot be upgraded.

It is expected that the vCenter upgrade would be blocked from moving forward and/or display a warning that performing the upgrade would break the compatibility of deployed clusters in the environment.

The best practice for upgrading a vSphere Supervisor environment is to check that all components will be compatible with the desired vCenter Server version before upgrading vCenter Server:

Workload Cluster TKR version
Supervisor Cluster version

How vSphere Supervisor Updates Work Documentation

Note: The Supervisor cluster and workload cluster TKR versions do not need to match for compatibility purposes.

Services and pods on the vSphere Kubernetes cluster will continue to work despite TKR incompatibility, however, the incompatible cluster cannot be managed, edited or upgraded.

VXRail environments may not block upgrades, which may result in incompatibility between the workload clusters and vCenter.

Resolution

In vSphere Supervisor, all workload clusters must run on a TKR version compatible with the environment.

The vCenter Server with TKR Interoperability Matrix can be used to determine Tanzu Kubernetes Releases (TKR) compatibility with vCenter versions.

A stranded workload cluster on an incompatible TKR version will need to be recreated on a compatible TKR version.

Back up and restore solutions can be used to migrate workloads off the incompatible workload cluster onto the newly created TKR compatible workload cluster.

Additional Information

After the TKG service supervisor service is installed, use the TKG Service with TKR Interoperability Matrix to compare TKR compatibility with the TKG service supervisor service.

Compatibility with the TKG service supervisor service overrides the TKR compatibility with the vCenter Server version

TKG service supervisor service upgrades that would cause workload clusters running on older TKRs to become incompatible with the desired TKG service supervisor service are prevented with a warning in the vSphere web UI.