Se_objsync using high cpu on Service Engine (SE)

Products

• VMware Avi Load Balancer

Issue/Introduction

• After upgrade to 31.2.x, the cpu usage significantly increases on Service Engine (SE) 
• se_objsync process consumes high cpu. To validate the same, you can run command : > show serviceengine <sename> cpu from the Avi Controller CLI. 

Environment

Avi LB version >= 31.1.x

Cause

• Starting with Avi Load Balancer version 31.2.1, the objsync feature is enabled by default. 
• The SE objsync feature is a common framework for sharing various object-types across SEs.
• Under specific edge cases, the End-of-Life (EOL) value tracked for a Key-Value (KV) pair can be updated concurrently by two independent SE-internal processes without adequate synchronization, producing an inconsistent EOL state. When this inconsistency is detected during object distribution, the SE attempts to correct it by redistributing the object, so it can repeatedly re-trigger the same detection-and-correction cycle. This results in continuous inter-process communication and sustained high CPU utilization on affected SEs.

Resolution

Workaround: 

Workaround 1: 

• We need to identify the VS/pool configuration responsible for triggering the issue.
• To assist with the same, we would require the SE tech-supports for the impacted SEs and debuglogs. Ref techdoc for log collection : collecting-tech-support-logs-using-avi-load-balancer-ui 
• Please reach out to Broadcom Support (Avi load balancer) team for analysis of the issue with the support bundles mentioned above.
• Once the VS/Pool is identified, remove the persistence profile associated with the virtual service and save it. 
• You can re-attach the same persistence profile to the pool again and save it. 

Workaround 2: 

• Disable se_objsync, if the above workaround does not help for long. 
• Login to cli of Avi controller leader using the command "shell" and  execute the below commands to disable objsync: 

> configure serviceenginegroup <segroup-name>

> objsync_mode objsync_disabled

> save

• Use the below command to validate the status of objsync 
  
   > show serviceenginegroup <segroup-name> | grep objsync_mode

Note: It is advised that once you upgrade to the fixed version, you should re-enable the objsync from the Avi CLI using the below commands. Before enabling, make sure port 4001 is allowed between the SEs. 

> configure serviceenginegroup

> objsync_mode objsync_enabled

> save

Fix: 

• Fix for this issue is targeted to be released in future version release of Avi LB software. 
  • Subscribe to this article to be updated on the fix status; see Subscribe to a Broadcom knowledge article by article or product