AVI deployments fail with "INTERNAL SERVER ERROR"

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

AVI deployments fail with  "INTERNAL SERVER ERROR".    vRA creates Avi Virtual Service objects one at a time. This is to reduce concurrency failures when using Avi controller version 22.x.

This issue is fixed by serializing the creation of Virtual Services. This will slow down concurrent deployments.

If you are using vRA with Avi controller version 30 and above, the creation of multiple Avi Virtual Service objects can be enabled to speed up completion of deployment requests.

Environment

VMware Aria Automation 8.x

VMware Avi Load Balancer 22.x

Cause

 In Avi 22.x there is a known issue when there are multiple concurrent requests made of blueprints that contain Avi resources. 

Resolution

Steps:

1. Log in to any vRA node

2. kubectl -n prelude edit deploy  tango-blueprint-service-app - Remove following JAVA OPT properties

-Dblueprint.execution.limitRules[0].name=serial-virtual-service-create
-Dblueprint.execution.limitRules[0].target=task
-Dblueprint.execution.limitRules[0].scope=global
-Dblueprint.execution.limitRules[0].concurrency=1
-Dblueprint.execution.limitRules[0].condition=data.taskType=='resource'&&data.resourceOperation=='create'&&data.resourceType=='Idem.AVILB.APPLICATIONS.VIRTUAL_SERVICE'

    3. Restart blueprint service:

         1. kubectl scale -n prelude --replicas=0 deployment/tango-blueprint-service-app

         2. kubectl scale -n prelude --replicas=n deployment/tango-blueprint-service-app

         Here "n" is number of nodes in the vRA instance

    4. Validate blueprint service is up and running for all expected nodes:

          kubectl -n prelude get pods | grep tango-blueprint-service-app

 

Note: These changes will be reverted in case the services are restarted using deploy.sh, for example, after an outage, and will also be reverted after an upgrade to a newer version of Aria Automation.