Orphaned Objects in Avi Load Balancer After vSphere Workload / Guest Cluster Deletion

Products

• VMware Avi Load Balancer

Issue/Introduction

When a vSphere Workload / Guest (VKS) cluster is deleted, the associated Avi Load Balancer (ALB) objects, including Virtual Services (VS), Service Engines (SEs), Service Engine Groups (SEGs), Pool, L4 Policy Set and Logical Router/Logical Switch (LR/LS) entries, are not removed gracefully.

This can lead to errors when creating subsequent VKS Workload / Guest clusters

Environment

 

• vSphere Kubernetes Service (vSphere Workload / Guest Cluster) Deployments.
• Avi ALB integrated with VKS vSphere Workload / Guest Cluster

 

Cause

Objects in VMware Avi Load Balancer created by AKO running in the guest/workload cluster cannot be deleted if the deleteConfig flag is set to false in the AKO ConfigMap.

By default, the deleteConfig flag in the guest/workload cluster's AKO ConfigMap is set to false. Upon cluster deletion, the VKS Supervisor attempts to delete the referenced object from Avi Load Balancer (T1 Router). However, because the Virtual Service VIP referenced by the T1 router, deletion is blocked.

Reference TechDoc Link

Resolution

To delete Workload / Guest Cluster reference object from Avi Load Balancer


Option 1:

     1.1 - Change the AKO Config Map field "delete_config" : True

     1.2 - Uninstall Workload / Guest Cluster AKO - that forced to delete all the reference object of that cluster from Avi Load Balancer 

     1.3 - Delete the Workload / Guest Cluster from vSphere Kubernetes Service (VKS)

To clean up the setup, edit the AKO ConfigMap and set deleteConfig: "true" to delete AKO-created objects in VMware Avi Load Balancer.

Once this flag is set, the annotation AviObjectDeletionStatus with the value Started is added to the AKO StatefulSet.

Note: An AKO reboot is not required when changing the deleteConfig flag value between true and false.

Option 2:

     2.1 - Delete the Workload / Guest Cluster from vSphere Kubernetes Service (VKS)

     2.2 - Delete the Workload / Guest Cluster's Virtual Service L4 Policy Set (L4 Policy Set Name start with Cluster Name example: "vks-xyz-dc1-prod--") from Avi Load Balancer UI > Templates > Policies > L4 Policy Set     

     2.3 - Delete the Workload / Guest Cluster's Virtual Service with reference object from Avi Load Balancer UI > Applications > Virtual Services > Select Virtual Services using checkbox and click on Delete button

     

Note: The Avi Load Balancer Object recursive-delete logic walks a specific list of child objects of Virtual Service and it considers safe/exclusive to clean up typically Pool, VS VIP if unshared.