Cluster Stuck in Deleting State

Products

• VMware Telco Cloud Automation

Issue/Introduction

Telco Cloud Automation (TCA) clusters are stuck in a "Deleting" state and cannot be successfully removed from the environment. The affected clusters become "orphan clusters" because they were previously deleted from TCA management, but residual data and artifacts (such as machines, vspheremachines, and vspherevms) still remain within the environment permanently stuck in a "Terminating" state.

Environment

3.x

Cause

Custom resources within the cluster's namespace have finalizers attached, and the controllers responsible for processing these finalizers are unresponsive. This prevents the garbage collection of the residual objects and their parent namespace.

 

Resolution

To clear the environment and fully remove the orphan cluster, manually clean up the residual data and remove the finalizers from the stalled objects:

1. Namespace Cleanup Setup: Ensure the namespace for the cluster exists (create it if necessary) so you can target the remaining artifacts correlated to that namespace.

2. Resource Removal: Delete all machine, vspheremachine, and vspherevm custom resources within the specified namespace:

   kubectl delete machines,vspheremachines,vspherevms --all -n <NAMESPACE>

3. Manual Finalizer Removal (Machines): If objects remain stuck in a Terminating state, manually patch them to remove the finalizers:

   kubectl get machines -n <NAMESPACE>-o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

   kubectl get vspheremachines -n <NAMESPACE>-o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

   kubectl get vspherevms -n <NAMESPACE>-o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

4. Manual Finalizer Removal (Cluster Objects): Target the remaining Cluster, TCA NodePool, and vSphereCluster objects within that specific namespace and remove their finalizers:

   kubectl get cluster,tcanodepool,vspherecluster -n <NAMESPACE> -o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

5. Delete the Namespace: Once all finalizers are cleared, delete the namespace to finish the cleanup:

   kubectl delete ns <NAMESPACE>