Supervisor cluster is incompatible due to vCenter update

Products

• VMware vCenter Server

Issue/Introduction

This document will have the steps to run to fix supervisor cluster incompatibility if you update your vCenter and ignore the warning, or have a 3rd party vendor product


When updating vCenter, a warning is typically displayed if the update may not be compatible with Supervisor Clusters for TKGs. However, customers using VxRail are not seeing this warning and are proceeding with the update to address a security issue.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The reason for this issue appears to be that the environment was either upgraded without acknowledging the warnings provided by vCenter during the pre-checks, or the warnings were suppressed by another product (e.g., VxRail).

Resolution

Check your supervisor cluster version and your workload cluster version before updating vCenter and apply and updates as needed before updating your vCenter.

Workaround:

1. Back up the current gcis-compatibility file
   
   kubectl get compatibilities -n vmware-system-ucs
   
   Find the current SV version:
   
   kubectl version
   
   Backup the compatibility document for that version
   
   kubectl get compatibilities -n vmware-system-ucs gciscompatibility-k8s-1.23.12 -o yaml > gciscompatibility-k8s-1.23.12.yaml
   
   
2. We will have to manually update the compatibility doc to remove the incompatibility for 1.22
3. Copy over the attached compatibility documents(new,old) to the SV cluster
   
   
4. Delete existing compat document corresponding to SV cluster
   
   kubectl delete compatibilities gciscompatibility-k8s-1.23.12 <--- This removes what is preventing the loading of the 1.22 Supervisor cluster
   
   
5.  Apply new compatibility document
   
   kubectl apply -f gciscompatibility-k8s-1.23.12-new.yaml <---- This file does not list 1.22 version so it will load the 1.22 supervisor cluster
   
   
6. Delete 1.22 TKr <--- You only need to run this if you run "kubectl get tkr" and see 1.22 in the list
   
   kubectl delete tkr v1.22.9---vmware.1-tkg.1.cc71bc8
   
   
7. Restart TKG Controller
   
   kubectl rollout restart deployment -n vmware-system-tkg vmware-system-tkg-controller-manager <---This should start the upgrade to 1.22 for the supervisor clusters
   
   
8. Once the TKr is recreated, upgrade all the clusters to 1.22 and then to 1.23.
   Verify TKr has been recreated.
   
   kubectl get tkr
   
   
9. Wait for all clusters to get upgraded.
   
   
10. Revert back the compatibility document to the original document
    
    kubectl delete compatibilities gciscompatibility-k8s-1.23.12
kubectl apply -f gciscompatibility-k8s-1.23.12-old.yaml
    
    
    
     

If for some reason you need to revert the settings:

Revert TKr compatibility : repeat Step#2, step#3.

 

Additional Information

If the update was initiated through VxRail, kindly follow up with Dell to ensure that future releases do not suppress warnings related to TKGs clusters.

Upgrading to a vCenter version that is too recent can break compatibility with the existing Supervisor Cluster version, preventing it from updating to the versions that align with the new vCenter version.

Do force upgrades if TKCs are going to become incompatible. We do not support such clusters.