Unable to create Load Balancer Objects in vSphere with Tanzu with NSX Advanced Load Balancer(Formerly known as AVI) on 7.0U3e or later
search cancel

Unable to create Load Balancer Objects in vSphere with Tanzu with NSX Advanced Load Balancer(Formerly known as AVI) on 7.0U3e or later

book

Article ID: 323440

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms

AKO pod fails to finish provisioning a load balancer object with this error.
 

<yyyy-mm-dd>T16:38:16.536Z        ESC[33mWARNESC[0m       rest/rest_operation.go:279      Error in rest operation is not of type AviError, err: Rest request error, returning to caller, *errors.errorString
<yyyy-mm-dd>T16:38:16.536Z        ESC[33mWARNESC[0m       rest/dequeue_nodes.go:220       key: admin/domain-c2014--ccams-test-test-control-plane-service, msg: got error while executing rest request: Error during POST: Rest request error, returning to caller, adding to slow retry queue


This issue only happens when you have more than the one Default Service Engine Group configured in AVI.

SE Groups can be viewed in AVI via: Infrastructure- > Service Engine Group 

Note: The example below shows 2 Service Engine Groups. If you only have 1 default group here, then you would not be running into this issue. 




Issue can happen either:

  • After updating VMware vSphere with Tanzu's Supervisor cluster to 7.0U3e

OR

  • While deploying VMware vSphere with Tanzu on or after 7.0U3e


You can correlate if supervisor version to VC version via the chart below.

The Supervisor version will look something like "v1.21.0+vmware.wcp.2-vsc0.0.12-18735554"
To correlate this to the chart below, the first part of the version is the k8's version noted in the 4th column, and the other one is the vcs0.0.x.  
 

Version Release Date vCenter build Supported K8 versions vSphere with Tanzu Version
vCenter Server 7.0 Update 3m 2023-06-22 21784236 1.23
1.22
1.21		 0.0.22
vCenter Server 7.0 Update 3L 
 		 2023-03-30 21477706 1.23
1.22
1.21		 0.0.22
vCenter Server 7.0 Update 3k 
 		 2022-02-23 21290409 1.23
1.22
1.21
1.20		 0.0.21
vCenter Server 7.0 Update 3j  2022-12-22 20990077 1.22
1.21
1.20		 0.0.20
vCenter Server 7.0 Update 3i  2022-12-8 20845200 1.22
1.21
1.20		 0.0.20
vCenter Server 7.0 Update 3h  2022-09-13 20395099 1.22
1.21
1.20		 0.0.19
vCenter Server 7.0 Update 3g  2022-07-23 20150588 1.22
1.21
1.20		 0.0.17
vCenter Server 7.0 Update 3f 2022-07-12 20051473 1.22
1.21
1.20		 0.0.17
vCenter Server 7.0 Update 3e  2022-05-12 19717403 1.22
1.21
1.20
1.19.1		 0.0.15
vCenter Server 7.0 Update 3d  2022-03-29 19480866 1.21.0
1.20.2
1.19.1		 0.0.13
vCenter Server 7.0 Update 3c  2022-01-27 19234570 1.21.0
1.20.2
1.19.1		 0.0.12
vCenter Server 7.0 Update 3a  2021-10-21 18778458 1.21.0
1.20.2
1.19.1		 0.0.12



Environment

VMware vCenter Server 7.0.x

Cause

Known issue with the AKO pod being unable to deploy lb objects when the backing AVI infrastructure has more than one SE Group. 

Resolution

This issue is resolved in 7.0U3k / Supervisor version 0.0.21. The code fix is in the AKO pod on the supervisor which means that you must update the supervisor cluster in addition to updating the VMware vCenter Server.

Workaround

Remove the extra SE Group so you only have one default SE Group.

Powered by Wolken Software