The Service Engine static IP configuration may be lost during periods of intense resource depletion

Products

• VMware Avi Load Balancer

Issue/Introduction

• This behavior occurs in environments configured as a no-access cloud utilizing static IP assignments for Service Engine data interfaces.
• When an SE disconnects and subsequently re-registers with the Controller, resource constraints on the host can impact performance.
• If the underlying host fails to provide adequate CPU and memory resources to the Controller—leading to a se_mgr process crash—the Controller may inadvertently issue a delete API call, removing the configured static IP from the SE.
• Below events can be seen on the controller for the same time period: Operations > Events > All Events

• Below redis errors can be seen in the se_mgr logs > /var/lib/avi/log/glog

• The SE's CPU availability can be viewed in the vCenter UI under Reported SE VM > Monitor > Performance > Advanced. Please see this article for more information on CPU readiness.

Environment

• VMware Avi Load Balancer
• No Access cloud - vCenter

Cause

• This behavior can occur due to transient Redis performance degradation driven by systemic CPU and resource constraints. When the underlying environment experiences heavy CPU contention:
  • The Redis server may suffer latency and return errors to calling processes (se_mgr, resmgr).
  • Redis access timeouts and lock acquisition failures can trigger unexpected service bail-outs, process restarts, or temporary recreation of Service Engine objects in memory.
  • Due to these processing delays and lock failures, the Controller's synchronization logic may fail to correctly re-apply the static IP upon SE reconnection.

Resolution

• Ensure that adequate CPU and memory resources are properly provisioned and reserved for your Controller and SE deployments to prevent system-wide resource starvation and minimize Redis latency.
• AV-284453: This issue will be addressed in future release of 32.2.x train. The fix introduces code enhancements including increasing MAX_LOCK_RETRIES in Acquire Locks and adding equivalent retry logic in Release Locks to provide sufficient tolerance for transient Redis slowness and prevent lock-related failures

Additional Information

• Contact Support for assistance with the same. Scroll to the bottom of the Contact Support and click on your respective region.
• Refer to Contact Broadcom Support for standard support procedures.
• Subscribe to this article to be updated on the fix status; see Subscribe to a Broadcom knowledge article by article or product