Delete cluster gets stuck if AVI is enabled

Products

• VMware

Issue/Introduction

Symptoms:
If AVI is enabled, and cluster deletion is stuck for more than 5 minutes. This could happen due to many reasons.

Cause

One of the common reasons that the cluster is stuck is that the AKO’s finalizer is blocking the cluster from being deleted. This could happen for many causes: 

1） AKO is not deployed successfully

2） AKO is not running correctly

 

The symptom is that the cluster object has the below condition set to false `AviResourceCleanupSucceededCondition`. 

 

kubectl get <cluster name> -n <namespace> -oyaml 

 

This finalizer is added intentionally because the cluster deletion should not succeed if there are stale resources left for this cluster.

Resolution

If  that cluster needs to be deleted then here is the procedure: 

1. Remove the finaliser from the cluster
   1. Kubectl edit cluster <cluster-name> -n <namespace>
   2. Under the `finalizers` section, remove the finalizer with key `ako-operator.networking.tkg.tanzu.vmware.com`
2. Delete the remaining resources of that cluster in AVI controller if there is any
   1. This is necessary since usually this happens when AKO is having trouble cleaning up resources in the AVI Controller. Then customer needs to manually pinpoint the resources of that cluster, resources contains but not limited to 
      1. Virtual service
      2. Server Pool
   2. The resources usually has name <namespace>-<cluster-name> pattern

Workaround: