Service Engine Creation may take a long time or fail completely in a scaled vCenter setup

Products

• VMware Avi Load Balancer

Issue/Introduction

• SE creation workflow, for SE Groups with cluster scoping enabled, is taking a long time in scoping clusters which is leading to multiple failures.
• You may see multiple SE creation failure events like below:
  

  Default-Group
  SERVICEENGINEGROUP
  CREATE_SE_FAIL
  N/A
  Service Engine Avi-se-<> creation failed
  Description
  Service Engine Avi-se-<> creation failed
  
  spawn_se_details
  se_name: Avi-se-<>
  se_cookie: xxxxxxxxxxxxxxxxxxxx
  host_name:
  availability_zone: N/A
  status_code: 11
  reason: Service Engine creation timed out
  vcpus: 4
  memory: 8192
  se_grp_name: Default-Group
  cloud_name: <>
  flavor_name: N/A

Environment

• Avi deployments with vCenter write access cloud and NSX-T clouds.
• SE Group has cluster scoping configured. 
• Environments where the target vCenter for SE creation has a large number of hosts/clusters. From our testing we have seen slowness in vCenters with 500 hosts, but it can happen at a scale lower than that as well.
• Avi versions 22.1.x-22.1.7-2p10, 30.1.x, 30.2.1-30.2.4, 31.1.1 

Cause

• With cluster scoping configured in the SE Group, the API calls to fetch the host and cluster details from the target vCenter where the SEs are to be deployed take a very long time.
  • Reference guide for cluster scoping configuration.
• Because of this, the internal timeouts for the SE creation kick-in resulting in multiple SE spawn timeout events. 
• Eventually, some creations may go through, but that behavior is not deterministic. 

Resolution

• The API calls to fetch the hosts/clusters from vCenter have been optimized from the product perspective.
• Bug ID: AV-245258
• Details: Service Engine creation may time out in scaled vCenter environments with many hosts, with cluster scoping enabled in the SE Group.
• Fix Versions: 22.1.7-2p11, 30.2.5, 31.2.1
• Release notes: 22.1.7-2p11, 30.2.5
• Workaround: 
  • Configure Host scoping in the SE Group instead of cluster scoping. 
  • With scaled setups, this operation may be difficult to perform on the UI. We would need to configure this via the CLI.
  • Follow the guide here for the steps to configure host scoping in the SE Group.