Avi Load Balancer virtual service unreachable in NSX-T cloud after static routes are deleted from Tier-1 gateway

Products

• VMware Avi Load Balancer

Issue/Introduction

• Avi LB VIP is not reachable from the same segment as that of the VIP
• We see events on Avi UI under operations > events > All events, with respect to NSX or vCenter accessibility issues
  
   
• Logs mentioning “Stale Route Deleted from Tier1” is seen in /var/lib/avi/log/cc_agent_go_<cloudname>.log on Avi Controller leader node. 

Environment

Avi Load Balancer integrated with NSX T cloud.

Cause

• Intermittent API timeouts from NSX-T and vCenter authentication failures caused Avi  cloud discovery to fail, which cleared internal caches.
• During a subsequent discovery cycle, the controller found no valid VRF contexts and mistakenly assumed existing static routes were stale, briefly deleting them from the Tier-1 gateway.
• This race condition caused the temporary VIP unreachability until API connectivity restored and the routes were recreated

Resolution

Workaround

To reduce the probability of the race condition recurring, increase the AVI controller's reconcile and discovery  timer:

1. SSH to Controller leader and login to cli using the command : > shell
   
   
2. Check the cloud_discovery_interval and cloud_reconcile_interval using the below command, by default, it is set to 5 minutes
   
   > show controller properties | grep cloud_
   
   
3. Increase the cloud_discovery_interval and cloud_reconcile_interval. Below are steps to be executed from Avi LB (cluster leader node) CLI:

 

> configure controller properties
> cloud_reconcile_interval 20
> cloud_discovery_interval 20
> save

While the above fix can help to reduce probablity of landing into the issue, the issue is essentially triggered if NSX-T or vCenter API timeouts are frequent in your environment. Hence, it is advised to investigate the underlying API latency or authentication failures separately with NSX and vCenter Team. To open a case with Broadcom, refer kb-142884 

Fix:

• This issue will be fixed in future maintenance releases of Avi Load Balancer. Subscribe to this Broadcom knowledge article to be updated about the fix.