Virtual Services exposed by Supervisor AKO (AKOO) gets deleted and re-created every 10 minutes

Products

• VMware Avi Load Balancer

Issue/Introduction

Users may observe the following behavior in environments  running with supervisor services exposed via Avi Load Balancer:

• LoadBalancer services (e.g., Contour) are deleted and recreated approximately every 10 minutes.
• The Virtual Service (VS) IP address (VIP) flaps between different values.
• The following error is present in the AKO logs on the Supervisor. To collect the AKO logs, refer: AKO logs
  

  RestOp method POST path /api/vsvip ... returned err {"code":0,"message":"map[error:Tier 1 cannot be derived from vrf]"}

Environment

Avi LB integrated with vCenter Cloud or NSX Cloud with vSphere Kubernetes Service

Cause

• This issue is caused by a conflict between the kapp-controller and the Supervisor Avi Kubernetes Operator (AKOO).
• The kapp-controller reconciles the svc-supervisor-management-proxy namespace every 10 minutes. During this cycle, it inadvertently strips the routing annotations required by AKO leading to the deletion and re-creation of the VS and as a consequence, IP address of the VIP changes. 
• Following log can be seen in AKOO logs: 
  

  RestOp method POST path /api/vsvip ... returned err {"code":0,"message":"map[error:Tier 1 cannot be derived from vrf]"}` 

 

Resolution

This is a known bug in the Supervisor and is targetted to be fixed in vCenter 9.1.1. Until then, refer the workaround below: 

1. Download the workaround script attached to this article: ako-overlay-workaround.sh.
2. Upload the script to the Supervisor.
3. Grant execution permissions:

   chmod +x ako-overlay-workaround.sh
   

4. Execute the script:

   ./ako-overlay-workaround.sh