Delete cluster gets stuck if AVI is enabled

Products

• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Symptoms:
If AVI is enabled, and cluster deletion is stuck for more than 5 minutes. This could happen due to many reasons.

Cause

One of the common reasons that the cluster is stuck is that the AKO’s finalizer is blocking the cluster from being deleted. This could happen for many causes: 

1） AKO is not deployed successfully

2） AKO is not running correctly

The symptom is that the cluster object has the below condition set to false `AviResourceCleanupSucceededCondition`. 

kubectl get <cluster name> -n <namespace> -oyaml 

This finalizer is added intentionally because the cluster deletion should not succeed if there are stale resources on AVI left for this cluster.

Resolution

If the cluster needs to be deleted then here is the procedure: 

1. Remove the finalizer from the cluster
   1. kubectl edit cluster <cluster-name> -n <namespace>
      
   2. Under the `finalizers` section, remove the finalizer with key `ako-operator.networking.tkg.tanzu.vmware.com`
      
      
2. Delete the remaining resources of the cluster in AVI controller if there are any:
   1. This is necessary since usually this happens when AKO is having trouble cleaning up resources in the AVI Controller. Then customer needs to manually pinpoint the resources of that cluster, resources contains but not limited to: 
      1. Virtual service
      2. Server Pool
   2. The resources usually have name <namespace>-<cluster-name> pattern