Manually patch ako operator addon secret with new AVI certificate in when upgrading TKG to 2.1(and above)

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

Customers will see the following error messages and the upgrade will fail if they:

• Enabled avi
• Rotated AVI certificate before upgrade
• try to upgrade TKG to 2.1.0(and above) 

avisession.go:666 Client error for URI: login. Error: Post “https://<avi ip|url>/login”: x509: certificate signed by unknown authority

Cause

This issue happens because customers rotated AVI certificate before upgrade but this new value won’t be updated in ako operator addon secret/tkg-pkg secret. When upgrading to a 2.1(and above) cluster, ako operator will be reconciled by tkg-pkg and AVI certificate value comes from value in tkg-pkg secret. Upgrade will fail because AVI still uses the old AVI certificate.

Resolution

Workaround:

 

Legacy cluster:

Before the upgrade, check AVI certificate in <cluster-name>-ako-operator-addon and AVI certificate in avi-controller-ca. If two values are different, manually patch  <cluster-name>-ako-operator-addon secret with the new AVI certificate.

$ kubectl get secret <cluster-name>-ako-operator-addon -n tkg-system

$ kubectl get secret avi-controller-ca -n tkg-system-networking

$ kubectl edit secret <cluster-name>-ako-operator-addon -n tkg-system

$ kubectl edit secret avi-controller-ca -n tkg-system-networking

 

Classy cluster

Before the upgrade, check AVI certificate in tkg-pkg-<namespace>-values and AVI certificate in avi-controller-ca. If two values are different, manually patch  tkg-pkg-<namespace>-values secret with the new AVI certificate.

$ kubectl get secret tkg-pkg-<namespace>-values -n tkg-system

$ kubectl get secret avi-controller-ca -n tkg-system-networking

$ kubectl edit secret tkg-pkg-<namespace>-values -n tkg-system

The issue will be fixed in Incheon.1

Additional Information

Modify the Avi Controller Certificates