How to migrate TKGI environment from one datastore to another datastore

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

This article provides steps for migrating TKGI environment from one datastore to another.

Environment

Tanzu Kubernetes Grid Integrated Edition 1.x

Cause

Resolution

TKGi v1.16+ supports a feature that will allow PersistentVolumes (PVs) used by TKGI clusters on vSphere to be migrated between datastores as described in Migrating Persistent Volumes Between Datastores Provisioning Support in Kubernetes. It is still possible to use Velero to carry out the migration on later versions of TKGi. 

NOTE: This procedure should only be followed if both datastores are in the same ESXi cluster and both datastores are accessible from the ESXi hosts where TKGi is running. Failing to adhere to this requirement could result in loss of persistent volumes. 

Procedure:

1. Take a backup using BBR and backup of Kubernetes persistent volumes using Velero. It is important to take a backup of both. Specially K8s persistent volumes. Changing the datastore in the tile does not directly migrate K8s persistent volumes and its data.
2. Change the datastore in persistent datastore and ephemeral store section of Bosh Director tile > "vCenter Config".
3. Select "Recreate Bosh Director VM" and "Recreate all VMs" deployed by bosh director option in Bosh director tile > "Director Config". Note: Do not select the Recreate Persistent Disk option.
4. Change the datastore under TKGI tile > "Kubernetes Cloud Provider".
5. Apply Changes with both the Director tile and the TKGI tile selected. Under the TKGI Errands deselect the upgrade all clusters errand. After the apply is finished, only the Bosh director and TKGI API VMs and persistent disks would have been moved to the new datastore.
6. To move each of the TKGI clusters, use the bosh cli and run bosh -d service-instance_<ID> recreate for each service-instance deployment. (Run bosh deployments --column=name | grep service-instance_ to get the list of deployment IDs.)
7. To move the OpsMan VM use a standard storage vMotion.
8. Finally we can migrate TKGi volumes to the new datastore using either Velero (Option A) or Cloud Native Storage migration on TKGi v1.16+ (Option B).
   1. Follow the official documentation to restore Kubernetes workloads to the new datastore using Velero.
   2. Follow the official documentation to migrate volumes to the new datastore using cloud native storage migration. 

Important Notes:

• When you recreate the clusters make sure you have backup copies of your apps.
• If you have single control plane node still you can proceed with the previous steps, however you will not be able to use kubectl commands for sometime until your control plane node is up.
• You might see the old stemcells vms still present on the old datastore after this process (powered off VM named sc-1234xx), From Vsphere GUI you can safely delete those old Stemcells, as the new stemcells will be automatically created on the new datastore.