Virtual Service Fault: "Virtual Service needs X Service Engines for High Availability, but only Y are available"

Products

• VMware Avi Load Balancer

Issue/Introduction

Some Virtual services were reporting Virtual service fault : "Virtual Service (vip_id 1) needs 3 Services Engines for High Availability. Currently only 2 Service Engine(s) available"

Note: The number of Service Engines required may vary depending on the Service Engine group configuration.

Environment

All Avi Load Balancer Versions

Cause

1. Scenario-1

• The Service Engine (SE) group was initially configured with the following settings:
  
  • Max Number of Service Engines: 2
  • Scale Per Virtual Service Minimum: 2
  • High Availability Mode: N+M (buffer)
  • All virtual services in this SE group were placed on a minimum of 2 Service Engines.
  • If the Scale Per Virtual Service Minimum setting is later changed to 3, but the Max Number of Service Engines remains set to 2, the virtual service will require a minimum of 3 Service Engines. Since only 2 are available, this fault is triggered.

    2. Scenario-2

• The SE group was initially configured with the following settings:
  
  • Max Number of Service Engines: 3
  • Scale Per Virtual Service Minimum: 3 
  • High Availability Mode: N+M (buffer)
• All the virtual services associated with this Service engine group are placed on minimum 3 service engines.
• If one Service Engine is manually deleted or becomes unavailable, and the administrator then updates the SE group settings to:
  
  • Max Number of Service Engines: 2
  • Scale Per Virtual Service Minimum: 2 
• These new settings apply only to newly created virtual services. Existing virtual services are not automatically scaled in and will continue to attempt placement on 3 Service Engines — triggering this fault.
  
  
  

Log Verification Section

Verify the below log from file res_mgr.INFO located in the directory of Controller Node  /var/lib/avi/log/glog 

I##### 08:35:35.940474  9561 vs_svc_api.cc:1199] T[#########] F[RmPlacementReadRsp] [vs-name:virtualservice-####:1] [rpc in] Read [##########-virtualservice:0<--resmgr_main_read_q] Cookie[#################] PlacementStatus:
  state[OPER_RESOURCES]
  reason[Reached configured maximum limit of 2 Service Engines in Service Engine group ####-####-###]
  reason_code[SYSERR_RM_MAX_SE_IN_GRP]
  num_se_requested[3], num_se_assigned[2]

Resolution

Resolution for Scenario-1:

• Increase the Max Number of Service Engines in the SE group to a value equal to or greater than the Scale Per Virtual Service Minimum setting.
  
  Navigation: Avi Controller UI → Infrastructure → Service Engine Groups → [Select SE Group] → Edit → Capacity & Limits

Resolution for Scenario-2:

• Manually scale in the affected virtual service to match the new SE group configuration.

1. Navigate to Applications → Virtual Services and locate the affected virtual service. 

   

    2. Hover over the virtual service name to reveal the options pop-up, Click Scale In.

    3. Don't select Any service engine at "Please select a Service Engine to scale in" option, proceed to click on the "Scale In" at the bottom. Click Scale In to confirm.

This will change the number of se requested for this VS to 2

 

Additional Information

• If the fault does not clear after performing the resolution steps, verify that the SE group settings have been saved correctly and that no other conflicting placement policies are in effect.
• For further assistance, contact VMware Avi Load Balancer Support via Broadcom Portal and provide the Controller debug logs and a screenshot of the current SE group configuration.