Service Engine crashes when DNS VS is Disabled/Enabled

Products

• VMware Avi Load Balancer

Issue/Introduction

Avi service engine hosting DNS VS can crash when DNS VS is disabled and enabled.

 

Environment

Applicable for GSLB setup on Avi Controller

Cause

• A stale GSLB object in the configuration caused the SE to crash.
• GslbService delete operations do not clean up downstream artifacts [VSGS table] resulting in dangling entries.
• Below is the stack trace generated for the crash

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/avi/bin/se_agent --log_dir=/var/lib/avi/log/glog --metrics_timer=60000 --r'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055d1b840673a in libse_agent_so_process_request (str=0x55d1ba0046c0 "\b\001\020\205\b\032\060gslbservice-xxxxxxxx-xxxx-yyyy-zzzz-xxxxzzzxxxxa\252@\317\004\n0n0gslbservice-xxxxxxxx-xxxx-yyyy-zzzz-xxxxzzzxxxxa\022\034testapi.abc.example.com\032\"testapi.abc.example.com\"\245\001\n#testapi.abc.exam"..., len=<optimized out>, mode=mode@entry=PRE, current_iteration=current_iteration@entry=0 '\000', total_iterations=total_iterations@entry=0 '\000') at /home/aviuser/workspace/22.1.4-2p1-Release/service_engine/agent/lib_se_agent_dataplane.c:3859
[Current thread is 1 (Thread 0x7faeb5852e40 (LWP 532372))]
#0  0x000055d1b840673a in libse_agent_so_process_request (str=0x55d1ba0046c0 "\b\001\020\205\b\032\060gslbservice-xxxxxxxx-xxxx-yyyy-zzzz-xxxxzzzxxxxa\252@\317\004\n0ggslbservice-xxxxxxxx-xxxx-yyyy-zzzz-xxxxzzzxxxxa\022\034testapi.abc.example.com\032\"testapi.abc.example.com\"\245\001\n#testapi.abc.exam"..., len=<optimized out>, mode=mode@entry=PRE, current_iteration=current_iteration@entry=0 '\000', total_iterations=total_iterations@entry=0 '\000') at /home/aviuser/workspace/22.1.4-2p1-Release/service_engine/agent/lib_se_agent_dataplane.c:3859
#1  0x000055d1b8209fa8 in SeAgentServiceImpl::se_agent_so_process_request (this=<optimized out>, req=<optimized out>, mode=mode@entry=PRE, current_iteration=<optimized out>, total_iterations=<optimized out>) at /usr/include/c++/9/bits/basic_string.h:2304

 

Resolution

Follow the below steps to delete the stale/dangling entries.

Steps to disable/enable the DNS VS
 a) Disassociate the DNS VS from the domain in the GSLB site Configuration and save.
 b) Disable the DNS VS.
 c) Associate the DNS VS to the domain in the  GSLB Site Configuration
 d) Wait for 60 seconds [This will take care of the NXDomain responses by DNS VS]
 e) Enable the DNS VS.

We have added a feature in 22.1.7-2p2 to detect the stale entry using API and CLI commands. Below is the usage.

Step 1: Stale entry check.

Using CLI

 shell
 gslb staleobjectchecks  

Using API

 Endpoint: /api/gslbsiteops/staleobjectcleanup/
 Method: GET

Step2: Stale entries cleanup

Using CLI

 shell
 gslb staleobjectcleanup

Using Api

Endpoint: /api/gslbsiteops/staleobjectcleanup/
Method: POST

Verification:
 Repeat the process in Step 1

Additional Information

https://vmw-jira.broadcom.net/browse/AV-218041