Traffic is not redirected to AVI FIP after AVI Service Engine Group has been migrated

Products

• VMware vDefend Firewall

Issue/Introduction

• You are running NSX-T and AVI Load balancer.
• You have configured Service insertion to be used with AVI.
  • Preserve IP has been configured from AVI.
• It is observed that when an SE group (Service Engine) has been migrated that traffic is no longer being redirected by service insertion.
• You may observe when switching between a working to the "problematic" SE that a POST API is not send by AVI to the NSX-T Manager to update the service insertion policy.
  • However, you will observe the POST API when switching to the working SE on the NSX-T Manager in /var/log/proxy/envoy_access_log.txt.

2024-09-02T15:26:39.652Z] <AVIcontrollerIP> <NSX ManagerIP> "POST" "/api/v1/serviceinsertion/services" "HTTP/1.1" 201 - 577 632 40 38 "10.76.106.7" "Go-http-client/1.1" "ce08504f-4fec-4819-8dc5-99812021cc5d" "nsxt01-manager.test.local" "127.0.0.1:7440"

• You may observe when switching from working to the "problematic" SE that a DELETE API is send from AVI to NSX-T manager in log file /var/log/proxy/envoy_access_log.txt.
  

2024-09-02T15:36:39.652Z] <AVIcontrollerIP> <NSX ManagerIP> "DELETE" "/api/v1/serviceinsertion/services/8b2ff997-2b70-4918-b1cd-1a81c7b4c115?cascade=true" "HTTP/1.1" 200 - 0 0 88 86 "10.76.106.7" "Go-http-client/1.1" "6f71c7a0-62d4-4386-93de-6d6c78a96d3a" "nsxt01-manager.test.local" "127.0.0.1:7440"

• • However, a similar DELETE API will not be seen when switching from the "problematic" to the working SE
• If you run the below API against the NSX-T Manager you may observe the "_last_modified_time" is not being updated when switching between the working and "problematic SE", indicating the virtual endpoint may be stale.

GET policy/api/v1/infra/tier-1s/<Tier-1 ID>/locale-services/<local-services ID>/endpoints/virtual-endpoints/

• You may observe when switching between the working and "problematic" SE that a service-insertion policy does not get removed when running the command get service-insertion on the NSX-T edge node as admin user.
• While running a trace flow in the NSX-T UI to simulate the return traffic it may be observed that the next hop is 0.0.0.0 when it should be the FIP configured in AVI.

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center.

VMware AVI Load balancer.

Cause

The traffic is no longer being redirected due to stale entries present on the NSX-T Manager from before the SE migration.

Resolution

All stale entries must be removed for the traffic to be redirected to the "problematic SE" .

Check if there is stale entries in the following areas and remove them:

• Virtual endpoints (API can be used to check this).
• L3SIpolicy (found in Corfu) ---> matches SI policy found on the edge.