Intermittent Service Engine Failures

Products

• VMware Avi Load Balancer

Issue/Introduction

Service Engines SE_DOWN events show the following failure reason: Avi Service Engine failed

Environment

Affects Versions: 22.1.3, 22.1.4

Cause

This was idetified as a known defect/bug where the Service Engine fails when a string group object attached to a Virtual Service updates with null/zero values.

Bug ID: AV-168720

Details: Stringgroup with no strings may lead to SE failure.

Service Engine Log Message Example:

The SE received and update from string group EXAMPLE at 07/29/2024 16:34:10 UTC from the controller.

se_trace.INFO

C255 16:34:10.000000 I0729 16:34:10.280057  2039 se_agent_dataplane_intf.cc:307] ## RPC Request[-13060306] Update STRINGGROUP[EXAMPLE]
C255 16:34:10.000000 I0729 16:34:10.280079  3225 se_agent_se_datastore_intf.cc:974] Handling done for notification uuid:  stringgroup-UUID version: 502959
C255 16:34:10.000000 I0729 16:34:10.280083  3225 se_agent_se_datastore_intf.cc:985] Stream version set to: 502959
C255 16:34:10.286373 [se_ag_str_grp_update_object:139] StringGroup update for EXAMPLE (stringgroup-UUID] PRE

However the update that was send by the controller had no key value entries (ie. empty string group) which is the root cause of the crashes.  The update made to the string group was removal of key value pairs, leaving the string group empty.

diff_sedatastore.log

2024-07-29T16:34:10.268Z   INFO    sedatastore/se_datastore.go:851 [Applying diff] {"diffVersion": "502959", "diffModelUuid": "stringgroup:stringgroup-UUID", "diffPayload": "{\"update\":[{\"before\":[{\"key\":\"/example/examples\"},{\"key\":\"/example/example\"}],\"to\":\"root.kv\"},{\"after\":3,\"before\":2,\"to\":\"root.configpb_attributes.version\"}]}"

 

Example Empty String Group:

{
  "url": "/api/stringgroup/stringgroup-uuid",
  "uuid": "stringgroup-uuid",
  "name": "EXAMPLE",
  "tenant_ref": "/api/tenant/?name=admin",
  "configpb_attributes": {
    "version": 3
  },
  "longest_match": false,
  "type": "SG_TYPE_STRING"
}

 

Workaround:

• Do not attach any empty key value string groups to virtual services.
• Do not edit and save any empty key value string groups (leaving them empty) that  are attached to virtual services.
• Remove any empty string groups from virtual services.

You can find empty string groups with the following jq command against your latest backup file: jq -r '.StringGroup[]|if(.kv == null) then .name else empty end' [back_up_file]

Resolution

Upgrade to the fix Versions: 22.1.6-2p4, 30.1.1, 30.2.1