Redistribution Loop Observed Due to Inconsistent Active/Standby Tags on Shared VIPs

Products

• VMware Avi Load Balancer

Issue/Introduction

• In this scenario, continuous Virtual Service (VS) redistribution events are observed between Service Engines (SE1 ↔ SE2).
• Frequent VS_SWITCHOVER events appear under Operations > Events, indicating repeated failover and rebalancing activity.
• This behavior results in:

  • Data-plane traffic disruption, due to frequent VS migrations between Service Engines.

Environment

The issue was observed in a setup where the Virtual Service (VS) is configured under a Service Engine Group with the following settings enabled:

• High Availability Mode: Active/Standby

• Distribute Load: Enabled

• Auto-redistribute Load: Enabled

Cause

1. The issue was triggered after creating a new Virtual Service "VS2".This VS shared the same VIP as an existing VS "VS1", but both had different active_standby_tag values:

• VS1 → ACTIVE_STANDBY_SE_1

• VS2 → ACTIVE_STANDBY_SE_2
  
  

  [admin:avi-local]: > show virtualservice VS1 | grep -i tag
  | active_standby_se_tag              | ACTIVE_STANDBY_SE_1                                 |
  
  
  
  [admin:avi-local]: > show virtualservice VS2 | grep -i tag
  | active_standby_se_tag              | ACTIVE_STANDBY_SE_2                                |

2. Shared VIPs must always use the same Active/Standby tag.This mismatch caused a placement conflict during the redistribution logic.

3. When placement of the new VS triggered redistribution, Avi attempted to rebalance VSes across SEs. Redistribution succeeded for few VSes but failed for the newly created        VS due to the tag mismatch.

4.This led to:

• Repeated removal and re-placement attempts for the problematic VS.

• Continuous redistribution cycles.

• Datastore congestion and eventual data-plane impact.

Resolution

Sequence of Events

1. A new VS (VS2) was created, sharing a VIP with another VS.

2. The new VS had a different active_standby_tag than the existing VS.

3. Placement triggered the Redistribution process.

4. Redistribution moved few VSes from SE1 → SE2 and reassigned tags.

5. The new VS failed placement due to the configuration mismatch.

6. Avi attempted to re-place and redistribute again, causing a loop.

7. Looping continued until one SE rebooted, triggering bail-out logic in the redistribution algorithm.

8. Post-reboot, stability was restored.

 

Recommendations / Preventive Measures

1. Configuration Consistency:
   Ensure all shared VIPs use the same active_standby_tag across all associated Virtual Services.
   This guarantees placement and switchover consistency.

2. Redistribution Control:
   Consider disabling the “Enable Auto-redistribute Load” option in the SE Group to prevent unnecessary redistribution loops during configuration changes.

3. Product Validation:
   Engineering will evaluate adding validation logic in future releases to prevent shared VIPs from being configured with mismatched tags.
   This check will be added in 31.3.1 Version.

 

Workaround:

Reboot the Service engine.