On VMWare deployments, interface route may remain in down state due to a race condition during SE bring-up leading to traffic disruption.

Products

• VMware Avi Load Balancer

Issue/Introduction

On VMWare deployments, interface route may remain in down state due to a race condition during SE bring-up leading to traffic disruption.

 

• During the issue occurrence time VS captures show that a '500 Internal Server Error' was sent to the client.

 

• VS logs display the error "server connection setup failed".

 

 

• On se_debug.info log indicates one of the cores is encountering a failure during ARP resolution.

C00 06:37:00.530201 [arpresolve:536] arpresolve: can't allocate llinfo for [10.x.x.x]
C00 06:37:00.530201 [ngx_event_connect_peer:194] *41 error [22] - connect() to 10.x.x.x failed
C00 06:37:00.530201 [ngx_http_upstream_finalize_request:5777] *41 Upstream Response: [10.x.x.x]:443 => [11x.11x.11x.xxx]:28527 => [10.x.x.x]:443 => [10.x.x.x]:51490 0 sent:0 recvd:0 r:0x5646cxxxxx c:0x5646xxxxx, upstream c:0x5646xxxxx

Environment

VMware

Cause

 

• The SE interface route remaining in the “Down” state which is the root cause for traffic failing with error response 500. 

show serviceengine se-xbtcm route

+-----------------+----------------+-----------+---------------+---------------------+
| IP Destination  | Gateway      | Interface  | Interface IP       | Route Flags              |
+-----------------+----------------+-----------+---------------+---------------------+
VRF 1
+-----------------+----------------+-----------+---------------+---------------------+
|   0.0.0.0/0     | 10.x.x.x     | eth9       | 10.x.x.x           | Up, Static,Gateway  |
| 10.x.x.x/24     | 0.0.0.0      | eth9       | 10.x.x.x           | Down, GWUp          |
| 10.x.x.x/32     | 0.0.0.0      | eth9       | 10.x.x.x           | Up, GWUp            |
+-----------------+----------------+-----------+---------------+---------------------+

• The interface route gets added when the IP is added. However, due to a race condition, the route can get added before the interface actually comes up. If this happens, the route is not marked Up. When the interface comes up, there is no update done on the route state.

Resolution

Temporary Workaround:

we can flap the interface by running the following commands from the namespace:

ifconfig avi_<eth_name> down
ifconfig avi_<eth_name> up

 

Permanent Fix:

Perform the upgrade to the following versions where the fix has been applied:

31.1.1
22.1.7
30.2.2
22.1.6-2p4