AKO Operator keeps reconciling and updating AVI users when AVI_LABELS is specified

Products

• VMware

Issue/Introduction

Symptoms:

If the AVI_LABELS is specified when creating the management cluster, the cluster admin will observe that AKO users keep getting updated on the AVI Controller dashboard. It results in DDOS-like behavior and high latency to access the AVI Controller, due to which the cluster admin can hardly create any load balancer type of service. If the cluster admin visited the Event page of the AVI Controller, continuous updates on the AKO user are observed.

Cause

Since TKGm 1.5, the AKO-Operator uses the skip-default label networking.tkg.tanzu.vmware.com/avi-skip-default-adc to indicate a cluster should not use the configuration of a default ADC.

It is caused by a logic bug in the AKO-Operator where two controllers keep adding & removing this skip-default label forever. The AKODeploymentConfig controller falsely assumes that a customized ADC always has non-empty clusterSelector, and that a default ADC has empty clusterSelector. But this assumption does not hold true anymore if the cluster admin specifies AVI_LABELS. 

Consequently, the controller mistakenly treated the default ADC as the customized one, and then keeps adding this label, causing it and the user object to be oscillating.

Resolution

This will be fixed in a future release. The workaround described below can be used to mitigate this issue in the meantime.

Workaround:

The cluster admin can work around this issue by removing this label from clusters and the cluster selector on default AKODeploymentConfig.

Pause the reconciliation for the AKO-Operator package.

kubectl patch pkgi ako-operator -n tkg-system --type "json" -p '[{"op":"replace","path":"/spec/paused","value":true}]'

 

Remove the cluster selector in the default AKODeploymentConfig CR

kubectl patch adc install-ako-for-all --type=json -p="[{'op': 'remove', 'path': '/spec/clusterSelector}]"

 

Remove the customized label of AVI_LABELS from every affected workload cluster.

kubectl label <WORKLOAD CLUSTER NAME> <the AVI_LABELS value>-

Example

kubectl label my-workload-cluster tkg.tanzu.vmware.com/ako-enabled=-

ako-operator package must remain in paused state in order for this change to remain persistent.

Additional Information

Configuring NSX Advanced Load Balancer in Workload Clusters