TKCs on TKR 1.27.x (and lower) in Ready False state After a Supervisor Cluster Upgrade on vCenter 8.0u3e or vCenter 8.0u3g

Products

• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

While on vCenter 8.0u3e or vCenter 8.0u3g, a Supervisor cluster upgrade was successful.

However after the Supervisor cluster upgrade, TKCs are in Ready False state.

These TKCs are running on TKR 1.27.X or lower

Environment

vCenter 8.0u3e or vCenter 8.0u3g

Supervisor cluster was upgraded while on either of the above vCenter versions

VKS automatically installed as version 3.3.2-embedded

VKS was not installed prior to the Supervisor cluster upgrade noted above

Affected TKCs are on TKR 1.27.X and lower

Cause

While on vCenter 8.0u3 versions, the VKS supervisor service is automatically installed (embedded) after a Supervisor cluster upgrade.

The installed VKS embedded version is incompatible with existing workload clusters running on TKR version 1.27 and lower.

On vCenter 8.0u3e or vCenter 8.0u3g, the Supervisor upgrade pre-checks do not warn about incompatibility of workload clusters after the Supervisor cluster upgrade when upgrading from a version before 8.0u3 Supervisor.

This can also happen if the Supervisor cluster was automatically upgraded through Supervisor Auto-Upgrade.

Resolution

Compatibility of workload cluster TKRs with the desired VKS version should be checked first before performing any upgrades.

See VKR and VKS Interoperability Matrix

Consult the VKR Upgrade Path matrix and VKR Release Notes for upgrade guidance:

VKR Upgrade Path Matrix

VKR Release Notes

 

For workload clusters that have encountered this incompatibility issue, checks will need to be made first to understand the state of the workload clusters on TKR 1.27 and lower.

1. Connect into the Supervisor cluster context
   
   
2. List all workload cluster TKCs with their versions and statuses:
   

   kubectl get tkc -A

   
   
   
3. Check if any of the above TKCs are in the middle of migration from wcp to vsphere objects:
   • See KB: How to Identify if a vSphere Kubernetes Cluster is Undergoing Migration from vSphere 7.X to vSphere 8.X
     
     
     
4. Confirm on the current VKS version:
   

   kubectl get pkgi -n vmware-system-supervisor-services
   
   NAME                          PACKAGE NAME           PACKAGE VERSION   DESCRIPTION
   svc-tkg.vsphere.vmware.com   tkg.vsphere.vmware.com   3.3.2-embedded   Reconcile succeeded

    

Open a ticket to VMware by Broadcom Technical Support with the above information to assess if a workaround is possible or only a redeployment can be advised.

Upload a Workload Management Support Bundle also known as wcp-support-bundle.

Additional Information

For TKCs on TKR 1.24.11, the only option is to redeploy as per the below KB article:

Unable to delete TKCs which have not migrated or TKC deletion is stuck