Unable to Upgrade Workload Cluster in vSphere Supervisor - denied the request: version upgrade not compatible with rules

Products

• Tanzu Kubernetes Runtime
• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to upgrade a workload cluster in vSphere Supervisor environment, the following error message is returned stating that the upgrade is not compatible:

error: tanzukubernetesclusters.run.tanzu.vmware.com "<workload cluster>" could not be patched: admission webhook "default.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: version upgrade not compatible with rules

Environment

vSphere Supervisor

This issue can occur regardless of the VKS supervisor service version installed.

This issue can occur regardless of whether or not the upgrading workload cluster is managed by Tanzu Mission Control (TMC)

Cause

Workload Cluster upgrades to a higher TKR or KR must be sequential, versions cannot be skipped.

For example, when upgrading from TKR v1.30 to desired TKR v1.32, the workload cluster must be first upgraded to TKR v1.31.

Once the workload cluster successfully upgrades to TKR v1.31, it can then be upgraded to TKR v1.32.

For more details, see How vSphere Supervisor Updates Work

Resolution

Workload cluster upgrades must be sequential. TKR versions cannot be skipped.

1. Confirm on the Updates Available for the workload cluster:
   For a workload cluster that uses a TKC:
   

   kubectl get tkc -n <namespace> <workload cluster>
   

   NAMESPACE      NAME              CONTROL PLANE      WORKER   KUBERNETES RELEASE NAME  AGE     READY   UPDATES AVAILABLE
   <namespace>    <workload cluster>      X               X     <TKR version A>          XXd     True    [<TKR version B>]

   For a cluster using cluster class without a TKC:
   

   kubectl describe cluster -n <namespace> <workload cluster> | grep UpdatesAvailable -B3
   
   - lastTransitionTime: "YYYY-MM-DDTHH:MM:SSZ"
   message: '[<TKR version B>]'
   status: "True"
   type: UpdatesAvailable
   
   

    

2. Check the VKR Release Notes and VKR Upgrade Path Matrix for the recommended next version:

   • VMware VKR Release Notes

   • VKR Upgrade Path Matrix
     
     
3. UpdatesAvailable is populated based on the next TKR version (N+1) that is in Ready and Compatible state in the environment.
   If this field is empty or does not contain the desired TKR version, see the following KB article:
   Unable to Upgrade vSphere Supervisor Workload Cluster - No Updates Available/Updates Available is Blank or Desired TKR is not Available due to Missing OsImage

Additional Information

After the VKS supervisor service is installed in vCenter, it overrides TKR compatibility in the environment.

In the vSphere web UI, the VKS service supervisor service version can be found under
Workload Management -> Services -> Tanzu Kubernetes Grid Service or vSphere Kubernetes Service

While connected to the Supervisor cluster, the VKS service supervisor service version can be found with the following command:

kubectl get pkgi -n vmware-system-supervisor-services

NAME                            PACKAGE NAME                PACKAGE VERSION              DESCRIPTION
svc-tkg.vsphere.vmware.com      tkg.vsphere.vmware.com      <vks service version>        Reconcile succeeded  

See the following documentation for details:

Interoperability Matrix between TKR and VKS service Supervisor Service

VKS service Supervisor Service Documentation