Virtual Service placement fails after upgrading to version 31.2.1 due to continuous Service Engine restarts

Products

• VMware Avi Load Balancer

Issue/Introduction

• After upgrading the controller to version 31.2.1, the Service Engines (SEs) started failing continuously.
• The SEs establish a successful connection with the controller; however, as soon as Virtual Service (VS) placement begins, the SEs reboot repeatedly, causing all associated VSs to flap (go up and down) continuously.
• From the SE console, the following broadcast message is observed, indicating a system-initiated reboot:

         

• SE Alert: SEs are observed going down repeatedly during VS placement.

         

• Event Logs: The event logs show a repeating sequence where the Service Engine transitions up and down, during which Virtual Services are added, removed, and re-associated with the Service Engine in a consistent and cyclical manner.

       

 

Environment

• Avi Load Balancer:  31.2.1

Cause

• In version 31.2.1, the handling of protobuf  annotations was modified.
  As a result, the SE graph DB now explicitly requests and downloads target objects during parent object fetch operations.
  
  
• Since this object (i.e certificatemanagementprofile) is not supported in the SE datastore model, the request fails and triggers a Resync and reboot, leading to continuous SE restarts.
• Logs on Service Engine:
• Path: /var/lib/avi/log/glog/ se_trace.INFO
  
• Logs on Controller:
• Path: /var/lib/avi/log/se_datastore.INFO

Resolution

• Apply Patch (Recommended)

  • Upgrade the environment by applying patch version 31.2.1-2p1. The patch 2p1 addresses the reported issue.

  • For additional information, refer to the 31.2.1 Release Notes
    Bug ID: AV-255249

• Rollback Option

  • If the patch cannot be applied, roll back to the previous stable version as an alternative mitigation.