Intermittent reachability issues for L4 load balancer services backed by AVI in VMware NSX
search cancel

Intermittent reachability issues for L4 load balancer services backed by AVI in VMware NSX

book

Article ID: 452014

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • You have two services running on a VKS cluster managed via VCF Automation (VCFA), backed by vSphere, VMware NSX, and an AVI Load Balancer.
  • These two services are of type L4 load balancer and they are intermittently not reachable every few minutes.
  • A traceflow confirms that when the intermittent issue occurs, traffic is sent via the default route north through the NSX edge instead of directly east/west to the destination host.
  • Checking route resolution on the host during the issue (Net-vdr -R -o resolve -e <source_IP> -i <destination_IP> <logical-router_UUID>) confirms that traffic resolves to the default route instead of the expected static route.
  • Checking the routing table on the host (get logical-router <router_ID> forwarding) from the NSX CLI when this issue occurs confirms the static route(s) are missing.
  • The var/log/proton/localhost_access_log.txt on the NSX manager shows frequent DELETE and PATCH API calls originating from the AVI controller to remove and recreate the static routes:
          var/log/proton/localhost_access_log.txt.17:1991:2026-08-06T14:49:26.320Z - "DELETE /nsxapi/api/v1/orgs/default/projects/<project_UUID>/vpcs/<vpc_name>/static-routes/<static_route-ID>HTTP/2.0" 200 - 56 55 +

          var/log/proton/localhost_access_log.txt.17:9995:2026-08-06T14:54:57.260Z - "PATCH /nsxapi/api/v1/orgs/default/projects/<project_UUID>/vpcs/<vpc_name>/static-routes/<static_route-ID>HTTP/2.0" 200 - 310 310 +

          var/log/proton/localhost_access_log.txt.15:4299:2026-08-06T15:10:29.575Z - "DELETE /nsxapi/api/v1/orgs/default/projects/<project_UUID>/vpcs/<vpc_name>/static-routes/<static_route-ID>HTTP/2.0" 200 - 58 58 +

    • The syslog on the NSX manager confirms the source of the API calls is the AVI controller IP:

          2026-08-05T22:32:53.562Z NSX 79265 - [nsx@4413 audit="true" comp="nsx-manager" entId="<static_route_ID>" level="INFO" org="default" proj="stes" reqId="<req_UUID>" request="DELETE /policy/api/v1/orgs/default/projects/<project_UUID>/vpcs/<vpc_name>/static-routes/<static_route_ID>" subcomp="manager" update="true" username="admin" vpc="<ID>"]           UserName="admin", Src="<IP_of_the_API_Source>", ModuleName="PolicyVpcStaticRoutes", Operation="DeleteVpcStaticRoutes", Operation status="success"

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

  • VMware NSX
  • AVI Advanced Load Balancer (NSX Advanced Load Balancer)
  • VCF Automation (VCFA)
  • VMware Kubernetes Service (VKS)

Cause

For NSX, this is expected behavior. Routes are being removed on the host when the static route deletion API is received on the NSX Manager from the AVI controller. The frequent DELETE and PATCH calls from the AVI controller cause the static routes to flap, intermittently forcing traffic to fall back to the default route (northbound) instead of the direct east/west route.

Resolution

Work with the AVI team to find the cause of the API calls deleting and recreating the static route in NSX. The AVI team must investigate the logic triggering the rapid DELETE and PATCH operations from the AVI controller.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following: NSX Manager support bundles, ESXi host support bundles, and the text of any error messages seen in the NSX GUI or command lines pertinent to the investigation.

Powered by Wolken Software