How to restore a single TKGi cluster when persistent disks for the cluster are lost.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

How to restore a single TKGi cluster when persistent disks for the cluster are lost.

Note: A few things to look out for before following the instructions:

• The Instructions section provides steps for the whole process (if an equivalent disaster event has already occurred and you have the backup, you can just follow the instructions for redeploying and restoring the cluster listed below):
  • creating a test cluster
  • creating its backup
  • creating a disaster scenario
  • redeploying the test cluster
  • restoring stateless workloads of the test cluster
• This procedure is only tested with TKGi clusters on the vSphere environment running flannel as its CNI plugin. It is recommended to try this procedure on a test cluster first (before trying to restore the damaged cluster) if you are on a different IaaS (for example AWS, GCP, vSphere with NSX-T etc)
• This procedure does not restore stateful workloads (workloads using PVs, load balancers, etc)
• Ops Manager/Bosh, Enterprise TKGi deployment states have not experienced any disaster event. If you run the following 2 commands you can see that the problematic cluster record is still available to TKGi as well as the Bosh director.

tkgi clusters

bosh -d service_instance-<GUID-Problematic-cluster> vms

Environment

Product Version: 1.7

Resolution

Note: These instructions are meant to be high-level in terms of explaining the procedure. It is assumed that you are familiar with the general procedure and know how to gather credentials required for the running the commands used in this procedure.

Step 1 - Create an Enterprise TKGi Cluster

Example (cluster1 - is the test cluster)

tkgi create-cluster cluster1 -e cluster1.api.pks.example.com -p medium -n 1

After the cluster is created, run the following to display Instance, Process state, VM CIDs, and Disk CIDs command:

bosh -d service-instance_<GUID> is --details --column=Instance --column=Process\ State --column=VM\ CID --column=Disk\ CIDs

Example output:

Instance                                           Process State  VM CID                                   Disk CIDs
apply-addons/#########-c207-4e78-a39d-############  -              -                                        -
master/########-06fc-4f29-af72-############        running        vm-########-5d44-4f7d-a4ec-############  disk-########-2060-40b2-a724-############
master/########-047d-4e04-bfbb-############        running        vm-########-8269-4c0e-a4e1-############  disk-########-15f7-41a4-bc8b-############
master/########-80db-46ef-8d8b-############        running        vm-########-ddc6-452a-b23e-############  disk-########-0969-4511-989b-############
worker/########-6b71-4108-9e3f-############        running        vm-########-6211-44bc-b35d-############  disk-########-bd38-4a7e-82bb-############

Step 2 - Create backup of the test cluster (cluster1)

Credentials for the following command can be gathered from here:
Prepare to Backup

pre-backup-check:

BOSH_CLIENT_SECRET='#####' bbr deployment --all-deployments --target #### --username ##### --ca-cert ##### pre-backup-check

backup:

Make sure to include --with-manifest option with the following command:

BOSH_CLIENT_SECRET=##### nohup bbr deployment --deployment service-instance_#### --target ###### --username pivotal-container-service-#### --ca-cert #### backup --with-manifest

Step 3 - Create a Disaster scenario

This is just a scenario which you can create to delete the persistent disks used by the test cluster. An equivalent event might have already happened in your environment in which case you can skip this step.

• From your IaaS console, power off the VMs of the TKGi cluster (cluster1). (Use VM CIDs obtained in step 1)
• Make sure that Bosh marks process state for each VM of the TKGi cluster (cluster1) as unresponsive agent
• For each Disk CID obtained in step 1 (Disk CIDs column), run the following command (this will make Persistent Disk of each VM to be marked as orphaned by Bosh)

bosh -d service_instance-<GUID> orphan-disk <Disk CID>

• Run a Bosh cleanup to delete the orphaned Persistent disks (this point will be considered a scenario where Persistent disks for cluster1 will be lost)

bosh clean-up --all

Step 4 - Redeploying the test cluster (cluster1)

From a previous backup or the backup created in Step 2, navigate to backup director (service_instance-<GUID-TIMESTAMP>) to use the manifest.yml for the following command to redeploy the test cluster (cluster1):

bosh -d service-instance_##### deploy service-instance_####_20200623T191343Z/manifest.yml --fix

This command will recreate the VMs of our test cluster (cluster1) and will also make sure that new persistent disks are provisioned and are attached to their corresponding VMs. After successful deployment, you will be able to access the test cluster using the TKG CLI (it might require you to login and get-credentials) but the cluster will not have any workloads running in it (kubectl get all -A) The next step will take care of restoring the stateless workloads for the test cluster.

Step 5 - Restore stateless workloads to the test cluster using a backup

This step only requires us to run the following BBR restore command to restore the stateless workloads:

BOSH_CLIENT_SECRET=#### nohup bbr deployment --deployment service-instance_##### --target ##### --username pivotal-container-service-#### --ca-cert xxxxx restore --artifact-path service-instance_####_####/

Where --artifact-path is the directory which contains the backup artifacts of the test cluster. 

After this operation completes, you can do use TKG CLI to do tkgi login & get-credentials again and then do a sanity check to make sure that workloads are restored (kubectl get all -A).