Manually delete left-over resourses in CaaS management cluster when a deployment has failed

Products

• VMware Telco Cloud Automation

Issue/Introduction

Unable to deploy management cluster as management cluster deployment is marked as failed. 

Environment

2.3

Cause

Some resources can't be deleted from bootstrap cluster, so Tanzu CLI reports errors. The management cluster deletion is stuck there, even if deletion operation is performed multiple times. Since the management cluster status is Failed, we need to manually delete leftover resources related with the management cluster.

Resolution

Left-over resources related with the failed management cluster needs to be manually deleted.

Workaround:

1. SSH into TCA-CP and delete the admin user Custom Resource context/minukube context
   kubectl get <crName> -n <namespace>
kubectl delete <crName> <managementCluster> -n <namespace>
2. Request force delete from TCA UI to delete the master cluster
3. When step2 finishes, switch to root user instead of admin, run the below command to delete data on TKG side. Like, it will delete MC data from `/root/.config/tanzu/config.yaml`, etc.
   tanzu config server delete <namespace>
4. Cluster nodes of management cluster has been deleted from vCenter? If not, please delete node manually from vCenter
5. Login to tca-cp VM, under admin user context/minukube context, check whether there are other resource are left in minikube cluster.
6. Directly query namespace of management cluster, if namespace exist, delete it.
   kubectl get ns <namespace>
kubectl delete ns <namespace>