AKO pod is not able to add static routes in Avi controller with error "Field check for static_routes failed : Unique constraints 'route_id' has duplicated value" or "Overlapping routes found"

Products

• VMware Avi Load Balancer

Issue/Introduction

• AKO pod is not able to add static routes in Avi controller with error "Field check for static_routes failed : Unique constraints 'route_id' has duplicated value" or "Overlapping routes found"
• The symptoms include

• • Backend pool members being marked down by health monitors due to improper routing. 
  • VS's being stuck in placement error "No network/subnet matched server IP x.x.x.x. Configure Pool placement settings with the required network/subnet. If the server is not directly connected but routable from the ServiceEngine, please also provide the static routes required to reach the server, before configuring the Pool."
• Under Operations>Events>Config Audit Trail, we should see a failed config_update event on vrfcontext
  
  
  
  
  

Environment

• AKO versions less than 1.14.1 and 1.13.2 releases.

Cause

• During kubernetes upgrade, worker Nodeip becomes unavailable for some-time leading to error in static route calculation/updation for that node.
• In such scenarios, AKO is not performing complete cleanup of internal cache object corresponding to that node which was leading to duplication of record.

Resolution

• Issue will be fixed in AKO 1.13.2 and 1.14.1 releases. 
• Workaround:
  • Reboot the AKO pod.
  • If there are VS's which are stuck in placement error state, disable/enable them after rebooting AKO pod.