Workload cluster residue remains in management CLI after deletion from TCA GUI

Products

VMware Telco Cloud Automation VMware Telco Cloud Platform

Issue/Introduction

Workload clusters appear to be deleted from the Telco Cloud Automation (TCA) GUI but remain visible or "Provisioned" in the Management Cluster.
Residual objects such as namespaces, clusters, machines, vspheremachines, and vspherevms persist when running kubectl commands against the Management Cluster CLI.
Subsequent cluster upgrades, lifecycle operations, or new deployments are blocked due to these stale or orphaned resources.
Virtual Machines (VMs) corresponding to the workload cluster nodes may still exist and remain powered on within the vCenter Server inventory.
This issue typically occurs when a deletion task is interrupted or fails to reconcile due to a network timeout, transient loss of communication between the Management Cluster and vCenter, or storage locks. As a result, the Cluster API (CAPI) and vSphere Provider (capv) controllers fail to remove the object finalizers, trapping the resources in an indefinite Deleting/Terminating state.

Environment

TCA 3.3

TCP 5.0.1

Cause

This issue typically occurs when a cluster is partially disassociated from the management layer before all underlying Kubernetes resources are fully cleaned up. Custom resources within the cluster's namespace often have finalizers attached. If the controllers responsible for these finalizers are unresponsive or the communication path is broken, the garbage collection process stalls, leaving 'orphan' objects and preventing the namespace from being deleted.

Resolution

WARNING: The following steps utilize "brute-force" finalizer removal. This tells the Kubernetes etcd database to forget about the resources without waiting for vCenter to destroy the actual infrastructure. You must perform the vCenter manual cleanup outlined in Step 5 to avoid orphaned VMs and resource leaks.

Step-by-Step Cleanup Procedure

Log in to the TKG Management Cluster.
Ensure the namespace for the cluster exists (create it if necessary) so you can target the remaining artifacts correlated to that namespace.

Bypass Cluster API and Bootstrap Webhooks

If the Core Cluster API (capi) or infrastructure (capv) controllers are actively blocking API deletion requests, temporarily remove the validating and mutating webhook configurations to clear the obstruction:

# Delete vSphere Provider Infrastructure Webhooks
kubectl delete mutatingwebhookconfiguration capv-mutating-webhook-configuration
kubectl delete validatingwebhookconfiguration capv-validating-webhook-configuration

# Delete Core Cluster API and Bootstrap Webhooks
kubectl delete validatingwebhookconfiguration capi-validating-webhook-configuration
kubectl delete mutatingwebhookconfiguration clusterbootstrap-mutating-webhook-configuration

(Note: These will automatically be recreated when the management controllers or cluster are redeployed/rebooted).

Strip Finalizers from Machine and Infrastructure Resources

Execute the following commands to clear the finalizers blocking the individual custom resources inside the workload cluster's namespace:

# Clear finalizers from core Cluster API and TCA node pool objects

kubectl get cluster,tcanodepool,vspherecluster -n <NAMESPACE> -o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

# Clear finalizers from specific VM and Machine constructs

kubectl get machines,vspheremachines,vspherevms -n <NAMESPACE>-o name | xargs -I{} kubectl patch {} -n <NAMESPACE> -p '{"metadata":{"finalizers":null}}' --type=merge

Verify Resource Eviction from the Management Plane
Before moving on to the namespace and secret deletion, run the following verification checks to ensure the etcd database has successfully evicted the stale tracking objects:
```
kubectl get cluster,tcanodepool,vspherecluster,machines,vspheremachines,vspherevms-n <NAMESPACE> 
```
Clear Remaining Core Secrets
Workload cluster bootstrap tokens and kubeconfigs will often hold onto their finalizers. Clear them with:
```
kubectl get secrets -n <NAMESPACE> -o name | xargs -I {} <NAMESPACE>  -p '{"metadata":{"finalizers":null}}' --type=merge
```

Delete the Namespace

kubectl delete ns <NAMESPACE>

If the namespace remains stuck in a Terminating state, bypass the namespace controller by pushing an empty finalizer array directly to the raw administrative API:

kubectl get namespace <NAMESPACE> -o json | \
tr -d "\n" | sed "s/\"finalizers\": \[[^]]*\]/\"finalizers\": []/" | \
kubectl replace --raw "/api/v1/namespaces/<NAMESPACE>/finalize" -f -

Manual Cleanup in vCenter Server (Mandatory)
Because the CLI bypasses infrastructure teardown, the physical VMs still exist in your environment.
1. Log in to the vSphere Client / vCenter Server.
2. Locate the folder or resource pool associated with the deleted workload cluster.
3. Select the stale worker nodes and control plane VMs.
4. Power Off the VMs, right-click, and select Delete from Disk.
Final Management Plane Cleanup
Once the workload dependencies and vCenter VMs are entirely cleared, return to the Telco Cloud Automation (TCA) GUI and proceed to delete or unregister the Management Cluster if you are performing a total system teardown. The GUI operation will now complete successfully without hanging.