How to Identify if a vSphere Kubernetes Cluster is Undergoing Migration from vSphere 7.X to vSphere 8.X

Products

VMware vSphere 7.0 with Tanzu VMware vSphere Kubernetes Service vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

This KB is intended to identify if a vSphere Kubernetes Cluster, also known as a Tanzu Kubernetes Cluster (TKC), workload cluster or guest cluster, is undergoing migration.

It will also advise against certain actions such as performing changes or upgrades to the migrating cluster.

Migration of all vSphere Kubernetes Clusters in a Supervisor cluster occurs after initiating an upgrade of the environment from vSphere 7.X to vSphere 8.X.

Environment

vSphere Supervisor environment that is upgrading from vSphere 7.X to vSphere 8.X

This issue can occur regardless of whether the affected migrating cluster is managed by Tanzu Mission Control (TMC)

Cause

Upgrades from vSphere 7 to vSphere 8 undergo an automatic migration of wcp objects into vsphere objects, and virtualmachineimages into clustervirtualmachineimages (cvmi).

The associated components are updated to reference the newly created objects for migration.

Documentation: "Upgrading from any vCenter Server release to any vCenter Server 8.x release"

Resolution

Upgrades from vSphere 7 to vSphere 8 undergo an automatic migration of wcp objects into vsphere objects and virtualmachineimages into clustervirtualmachineimages (cvmi).

This results in a rolling redeployment of all workload clusters in the Supervisor cluster to facilitate this migration.

IMPORTANT: While a cluster is migrating, it is important to not make any changes to the cluster or perform deletions of migrating objects:

Editing a migrating cluster can cause further issues during the migration and cause the migration of the cluster to become stuck.
It is advised to not make any configuration changes to the migrating cluster. This can prevent the cluster's migration from completing.
Please do not start an upgrade of the migrating cluster. This will cause the cluster's migration to become stuck.
Do not perform deletions of objects to force the migration. This will lead to permanent loss of those migrating objects.

Please see the below for steps on how to identify a cluster that has not yet finished its migration.

Connect to the Supervisor cluster context

Identify all workload clusters with the migrate-tkc label:

kubectl get tkc -o yaml -n <cluster namespace> | grep -i migrate

Check regarding the migrating cluster's progress of wcp object migration to vsphere object equivalents:
- NOTE: wcp objects being present may indicate that the migration is yet to complete for the corresponding vSphere Kubernetes cluster(s).
```
kubectl get wcpcluster,wcpmachine,wcpmachinetemplate -n <cluster namespace>
kubectl get vspherecluster,vspheremachine,vspheremachinetemplate -n <cluster namespace>
```
  - These wcp objects may still be present even if the corresponding TKC object does not have the "migrate-tkc" label.
    - This may indicate an issue with the system properly cleaning up the leftover wcp objects.
  - It would be best to reach out to VMware by Broadcom Technical Support referencing this KB article for help in cleaning up any leftover migration objects.
    - This is to avoid deletion of objects that are still in use or not yet fully migrated.
Confirm on the status of the migration of virtualmachineimage migration to clustervirtualmachineimage (cvmi):
- There should be equivalent clustervirtualmachineimages (cvmi) to the virtualmachineimages:
```
kubectl get virtualmachineimages -A
kubectl get cvmi -A
```
Check if there are any machinesets that are still referencing a wcpmachine and have a non-zero replica count:
- Search all machinesets referencing a WCPMachineTemplate.
  - If the migration completed successfully, there should be no machinesets referencing a WCPMachineTemplate and this output should return nothing:
```
kubectl get machineset -n <cluster namespace> -o yaml | grep -i -A2 wcpmachinetemplate
```
- Find all machinesets referencing a vSphereMachineTemplate. WCPMachineTemplates are expected to have been migrated to this:
```
kubectl get machineset -n <cluster namespace> -o yaml | grep -i -A2 vspheremachinetemplate

     kind: vSphereMachineTemplate
     name: <vSphereMachineTemplate name>
     namespace: <cluster namespace>
```
- List all wcpmachinetemplates and vspheremachinetemplates in the namespace which can be compared to the previous two outputs:
```
kubectl get wcpmachinetemplate,vspheremachinetemplate -n <guest cluster namespace>
```
The corresponding machinedeployments (md) and kubeadmcontrolplanes (kd) for the workload cluster can also be verified regarding referencing any WCPMachineTemplate:
```
kubectl get md,kcp -n <cluster namespace> -o yaml | grep -i -A2 wcpmachinetemplate
```
- And similarly verified that the machinedeployments reference a vSphereMachineTemplate instead, indicating that the migration was successful:
```
kubectl get md,kcp -n <cluster namespace> -o yaml | grep -i -A2 vspheremachinetemplate
```