Manually delete left-over resourses in CaaS management cluster when a deployment has failed

Products

• VMware Telco Cloud Automation

Issue/Introduction

Unable to deploy management cluster as management cluster deployment is marked as failed. 

You may get this message on the TCA UI:

Unable to create cluster ClusterName Reason: Cluster has exist in /root/.config/tanzu/config.yaml, please check! If this is invalid cluster data left before, manually delete it.Login to TCA-CP or Collect TechSupport Bundle of TCA-CP for the logs.

Environment

2.3

Cause

Some resources can't be deleted from bootstrap cluster, so Tanzu CLI reports errors. The management cluster deletion is stuck there, even if deletion operation is performed multiple times. Since the management cluster status is Failed, we need to manually delete leftover resources related with the management cluster.

Resolution

Left-over resources related with the failed management cluster needs to be manually deleted.

Workaround:

1. SSH into TCA-CP and delete the admin user Custom Resource context/minukube context
   kubectl get <crName> -n <namespace>
kubectl delete <crName> <managementCluster> -n <namespace>
2. Request force delete from TCA UI to delete the master cluster
3. When step2 finishes, switch to root user instead of admin, run the below command to delete data on TKG side. Like, it will delete MC data from `/root/.config/tanzu/config.yaml`, etc.
   tanzu config server delete <namespace>
4. Cluster nodes of management cluster has been deleted from vCenter? If not, please delete node manually from vCenter. If you have any instance of bootstape VM for this cluster that also needs to be deleted.
5. Login to tca-cp VM, under admin user context/minukube context, check whether there are other resource are left in minikube cluster.
6. Directly query namespace of management cluster, if namespace exist, delete it.
   kubectl get ns <namespace>
kubectl delete ns <namespace>