SE_OBJSYNC_PEER_DOWN event seen on the controller

Products

• VMware Avi Load Balancer

Issue/Introduction

You see the following event on the controller under Operations>Events>All Events:

Environment

All Environments

Cause

• The SE in the Avi Load Balancer hosts multiple virtual services to serve a specific application. A single virtual service can be scaled across several Service Engines (SE), and each virtual service comprises several objects that are created, updated, and destroyed. Some of these virtual service objects need to be available across all the SEs to ensure consistent operation across a scaled-out application instance.
• Earlier the Controller was used to distribute this information across the participating SEs. Each SE has a local REDIS instance that connects to the Controller, and the objects are allocated and synchronized across the SEs through the Controller. This scheme has limitations on scale, convergence time, and so on.
• Starting with 31.2.1, the SEs perform this distribution and synchronization without the involvement of the Controller. The inter-SE data distribution such as IP persistence, cookie persistence, key: value persistence and more will be done over a new infra termed ObjSync.
• This ObjySync feature will get auto-enabled for existing and new SE-Groups once the controller is upgraded to 31.2.x and above. The default port used for this communication is port 4001 which can be configured through objsync_port  in Service Engine group properties. This port needs to be made open between SEs. 
• If the port 4001 is not allowed in your environment such that the Service Engines cannot communicate over this port, persistence will be impacted and an event will be generated on the controller: "SE_OBJSYNC_PEER_DOWN".   

Resolution

• If you don't want to use this feature because of internal security postures and would prefer to stick to the legacy implementation, please follow the CLI steps below to disable it.
  
  

> configure serviceenginegroup <se-group-name>
serviceenginegroup> objsync_mode objsync_disabled
serviceenginegroup> save

• If you choose to continue using this feature, which is the recommended way, then please ensure port 4001 is accessible in your environments to allow inter-SE communication.

Additional Information

Please refer the below techdocs for more detail on the objsync feature:
Storing Inter-SE Distributed Object