Virtual service is not scaled out correctly

Products

• VMware Avi Load Balancer

Issue/Introduction

Virtual Service is not scaled out to the expected number of SEs. 

On the virtual service, you will see a fault :  Virtual Service (vip_id1) needs 3 Service Engines for High Availability. Currently only 2 Service Engine(s) available. 

Environment

vCenter cloud

Avi Load Balancer version 21.x

Avi Load Balancer version 22.x where existing SEs are upgraded from 21.x

Cause

The logs /var/lib/avi/log/glog/res_mgr.INFO suggest that SE did not have any available free interfaces 

However, looking at SE interfaces on UI, you do see free interfaces. 

This can happen if "is_avi_internal" flag is set to false for the free interfaces. This can be verified using the command "show serviceengine <se-name>" , scroll down to free data nics and check for "is_avi_internal" flag for the same. Reference screenshot below : 

Note : The above flag is important for versions - 21.x and early 22.x (upgraded SEs) where avi_internal used to be the default network for any unused SE data interfaces . Starting 22.1.1, Avi internal portgroup is not used for the newly deployed SEs. Refer Broadcom techdocs : Considerations for vCenter Cloud while upgrading from releases prior to 22.1.1

Resolution

Workaround to mitigate the issue is to disable the problematic SE and scale out to a new SE.