Virtual Service Failure with "Pool Update Failed" Fault

Products

• VMware Avi Load Balancer

Issue/Introduction

• A Virtual Service (VS) may enter a fault state on a standby Service Engine (SE), displaying a "Pool Update Failed" error. 
• This issue is noticed when standby Service Engine (SE) becomes Primary

Environment

Avi affected versions: <=30.2.7, <=31.2.2, 32.1.1

Cause

This fault can be triggered by two specific sequences of events related to pool server management and health monitoring settings.

Scenario1

1. A server is disabled with the server_disable_type set to ALLOW_NEW_CONNECTION_IF_PERSISTENCE_PRESENT. 
2. Health Monitors (HMs) are active on the primary SE but not on the standby SE. 
3. The server goes down on the active SE due to HM failure but remains up on the standby SE, which has no HMs. 
4. The disabled server is deleted from the pool.
5. A subsequent pool update will cause the VS to enter a fault state on the standby SE.

(or)

Scenario2

This case can happen even without the specific server_disable_type setting mentioned above.

1. Health monitoring on standby (hm_on_standby) is disabled. 
2. A pool is created and linked to a VS with a server that is already in a disabled state. 
3. The disabled server is subsequently deleted from that pool. 
4. Any future pool update will trigger the fault state on the standby SE.

Note: This issue does not occur if a server is enabled when the pool is created and is disabled at a later time.

Resolution

Workaround

• Enable Health Monitoring: Ensure that health monitoring is enabled for pools on the standby SE. 

  # Validate whether hm_on_standby knob is set to True or False for Service Engine Group
  [admin:#.#.#]: > show serviceenginegroup <Name of Service Engine Group>
  
  # If it's set to false, Enable it with below commands
  [admin:#.#.#]: > configure serviceenginegroup <Name of Service Engine Group>
  [admin:#.#.#]: serviceenginegroup> hm_on_standby
  [admin:#.#.#]: serviceenginegroup> save

• Enable Servers Before Deleting from Pool: Always enable a server before deleting it from a pool. Do not perform a delete operation on a disabled server.
• Avoid Disabled Servers on Pool Creation: Do not create a pool with a server that is already disabled.

Fix

• Fix for this issue is targeted to be released in future version release of Avi LB software. 
  • Subscribe to this article to be updated on the fix status; see Subscribe to a Broadcom knowledge article by article or product