Default AVI user for workload cluster didn't get created - Workaround for using AVI controller 21.1.x on TKGm 1.5

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
When creating workload cluster using TKGm 1.5 using AVI to provide control plane VIP, you may notice the AVI user for this workload cluster didn't get created when you are using AVI controller 21.1.x.

Cause

The problem caused by the AVI controller version didn't get specified in management cluster AKO Operator deployment, so the AKO Operator will use the default lowest support version to send request to AVI controller, which version is not supported by AVI controller 21.1.x any more.

Resolution

1. Switch to your management cluster kubeconfig context.

2. Pause ako-operator package reconcile:

kubectl patch pkgi ako-operator -n tkg-system --type "json" -p '[{"op":"replace","path":"/spec/paused","value":true}]'

3. ​​​​​Set ako-operator deployment AVI controller version environment variable, take AVI controller 21.1.3 as an example: 

kubectl set env deployment/ako-operator-controller-manager avi_controller_version="21.1.3" -n tkg-system-networking

4. Restart AKO Operator pod:

kubectl delete pod ako-operator-controller-manager-<random-hash-string> -n tkg-system-networking