Migrating VMs/Volumes across Datastores in vSphere with Tanzu / vSphere Kubernetes Supervisor

Products

VMware vSphere Kubernetes Service vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

vSphere with Tanzu / vSphere Kubernetes Supervisor contains the following storage locations that need to be considered when moving datastores.

Supervisor Cluster:

1. Supervisor Control Plane VM vmdks.
2. Supervisor Cluster's PVCs

3. Supervisor Services without PVCs

Guest Cluster:

4. Guest Cluster's controller and worker node vmdks. (and associated Volume Mounts if they are defined on the cluster)
5. Guest Cluster's PVCs.

Environment

vSphere Supervisor

Resolution

Use the following instructions to move each component one at a time. It does not matter in which order they are performed.

Prerequisite: Must have a storage profile for the destination datastore separate from the source. Both storage profiles must be attached to the supervisor namespace on which migration will be performed.

1. Supervisor Control Plane VM vmdks.

Currently the only way to move Supervisor Control Plane VM VMDKs safely:

- 1. Ensure that the Supervisor cluster version is on the latest version available according to the vSphere web client.
    - Each vCenter version supports three sequential Supervisor cluster versions. It would be best to be on the highest version possible for the current vCenter version.
  2. Confirm that both the new storage policy associated with the destination datastore and the old storage policy associated with the source datastore are both tagged and compatible for both datastores.
  3. Check that the destination datastore has enough space for the migration.
  4. Please open a ticket to VMware by Broadcom Technical Support referencing this KB article to have them run though the process with you.

2. Supervisor Cluster's PVCs

This is in reference to any PVC's that you are using that are NOT being used by any of the guest clusters. For example PVCs that you have attached to vSphere PodVMs on a supervisor namespace.

NOTE: All PVC's that relate to guest clusters will be moved in step 4.

Backup/restore these PVCs via the Velero Supervisor Service:

Follow the Velero Documentation to backup the PVCs and then restore them to the storage class that uses the new datastore.

Velero Documentation for Restoring a PV and PVC to a new StorageClass

3. Supervisor Services

This is specifically Supervisor Services using vSphere PodVMs which run on ESXi hosts and use a storage policy for datastore placement.

As these steps involve the direct deletion of system objects and causes downtime of the Supervisor service to be migrated, reach out to VMware by Broadcom Technical Support referencing this KB.

4. Guest Cluster Node vmdks

This is done by editing the TKC/Cluster object from the supervisor and changing the storage class from the storage class containing the source datastore to the storage class containing the destination datastore.

- IMPACT: This WILL trigger a cluster rollout as new nodes will be provisioned on the new storage and old nodes will be deleted.
  1. Confirm that new storage classes were created for the destination datasource for each storage class used in the environment.
  2. Add the new storage class(es) for the destination datastore to the namespace:
    - Workload Management -> Namespaces
    - Select the namespace to edit
    - Navigate to the namespace's Storage tab
    - Under Storage Policies view, click on the Edit button in the top right
    - Check the box for the new storage policy associated with the destination datastore.
    - Ensure that there is both the old storage class(es) and the new storage class(es) listed selected for the namespace.
  3. (Optional) It is advised to create a small test cluster using the new storage class(es) for the destination datastore to confirm that there will be no issues.
  4. From the Supervisor cluster context, edit the cluster to migrate by changing all storage class entries to the new storage class equivalent on the destination datastore:
    - Note: This includes editing the storage class for any volume mounts in the cluster.
    - Edit Cluster StorageClass TechDocs
  5. Allow for the edited cluster to complete its rolling redeployment of its nodes on the new storage class(es) for the new datastore.

5. Guest Cluster PVCs

The only way to migrate PVCs between datastores in vSphere with Tanzu is by backing them up and restoring them to the new storage class.

Make sure to involve the application team for concerns about in-flight data and how that should handled specific to the applications using these PVCs.

This can be done via any kubernetes backup/restore provider that has validated their product against vSphere with Tanzu.

Velero Documentation for Restoring a PV and PVC to a new StorageClass

Clean Up

Once all four components have been migrated to the new storage, the old storage profile can be removed from the namespace.

Under Workload Management under each namespace, Edit Storage can be used to clean up unused storage policies.

Check if the Content Library used by the namespaces needs to be migrated to the new datastore as well.

Under Content Libraries, the Summary tab for the selected Content Library will show the datastore it is on.
At this time, a content library cannot be moved across datastores. A new content library must be created on the new datastore.
All namespaces using the previous content library must then be pointed to the new content library on the new datastore.
- This can be done through Workload Management -> Supervisor name -> Configure -> Supervisor -> General -> Tanzu Kubernetes Grid Service -> Content Library -> Edit

Validate that the storage profile has been removed by verifying that the storage class objects for it on the Supervisor Cluster are no longer present:

Connect to the Supervisor cluster context
- ```
kubectl get sc -A
```

The old storage policy associated with the source datastore is no longer referenced by any objects in the environment.