Service Engine may crash when a DNS VS is listening on the same port for both UDP and TCP.

Products

• VMware Avi Load Balancer

Issue/Introduction

Service Engine may crash when the following requirements are met:

• A DNS VS with application profile type of DNS is placed on the Service Engine.
• The DNS VS is listening on the same port for both UDP and TCP. 

Environment

All versions running with the above VS config. 

This particular problem is not applicable to non-DNS L4 Proxy or L7 Proxy

Cause

When the Service Engine tries to send a large DNS response(received from the backend server) to the client on a TCP connection that is recovering from network drops

Resolution

The stack trace will include the function: panic (It should be present in initial #0 method calls)

Sample stack trace:

To investigate further, you can review the latest stack traces from the Controller or SE by accessing the following path:

CLI:

Login to Controller via ssh and run this command.Please note you have to replace the name of se_dp file here.

root@<Controller ip>:#  cat /opt/avi/archive/stack_traces/<se_dp.timestamp>.stack_trace
 
UI:
Navigate to Administration>Support>Crash Reports>Expand the latest crash file.

 

Workarounds:

1. Change the application profile type of DNS VS to System-L4.
   
                          OR

2. In the DNS VS there would be 2 services, one with UDP-per-Pkt and other with System-TCP-Proxy.
   Remove the service of type System-TCP-Proxy, create a new VS which shares the VIP with the DNS VS. Configure this new VS with application profile as System-L4 and System-TCP-Proxy network profile. 

With workaround 1, we will miss DNS information under the VS logs as the VS is configured with System-l4. 
With workaround 2, we will miss DNS over TCP related logs. 

 

The crash is fixed in 30.2.4, 31.1.2, 31.2.1.