Issues caused by the se_objsync feature on VMware Avi Load Balancers

Products

• VMware Avi Load Balancer

Issue/Introduction

- The SE objsync feature is a common framework for sharing various object-types across SEs.
- The main functions include, but are not limited to, building and maintaining the SE-nodes map, assisting in initial sync when the SEs comes up and syncing various tables across the SEs.
- However, the SE objsync feature has caused multiple issues in customer environments mainly with SE memory exhaustion and breaking persistence.
- These issues include:
-- Failure in Objsync connection over management interfaces between SEs might lead to memory exhaustion.
-- High CPU usage on the SEs due to channel enqueue failures in Objsync.
-- SE Persistence table with different client_ip to server_ip mapping for the same source client IP.

Environment

- Avi controller with se_objsync enabled in the ServiceEngine Group.

Resolution

- The current recommendation is to disable the se_objsync feature.
- With this the SEs will fall back to syncing the tables using the legacy method, via redis on the controller. There is no impact to the current functioning of the product.
- Please follow the steps below to disable se_objsync on the SEs:

1. ssh to the controller leader node with the admin user and login into shell
2. execute the following commands:
  > configure serviceenginegroup <se_group_name>
  > no use_objsync
  > save

- We will also need to reboot the SEs once this has been configured. Also, note that the SEs can be rebooted in a rolling manner. 
- This feature has been disabled by default starting v30.2.1.