Manually patch ako operator addon secret with new AVI certificate in when upgrading TKG to 2.1(and above)

Products

• Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:

Customers will see the following error messages and the upgrade will fail if they:

• Enabled avi
• Rotated AVI certificate before upgrade
• try to upgrade TKG to 2.1.0(and above) 

 

avisession.go:666 Client error for URI: login. Error: Post “https://10.242.40.240/login”: x509: certificate signed by unknown authority 

 

Cause

This issue happens because customers rotated AVI certificate before upgrade but this new value won’t be updated in ako operator addon secret/tkg-pkg secret. When upgrading to 2.1(and above) cluster, ako operator will be reconciled by tkg-pkg and AVI certificate value comes from value in tkg-pkg secret. Upgrade will fail because AVI still uses the old AVI certificate.

Resolution

The issue will be fixed in Incheon.1

Workaround:

If its Legacy cluster
Before the upgrade, check AVI certificate in <cluster-name>-ako-operator-addon and AVI certificate in avi-controller-ca. If two values are different, manually patch  <cluster-name>-ako-operator-addon secret with the new AVI certificate.

 

$ kubectl get secret <cluster-name>-ako-operator-addon -n tkg-system
$ kubectl get secret avi-controller-ca -n tkg-system-networking
$ kubectl edit secret <cluster-name>-ako-operator-addon -n tkg-system

If it is Classy cluster

Before the upgrade, check AVI certificate in tkg-pkg-<namespace>-values and AVI certificate in avi-controller-ca. If two values are different, manually patch  tkg-pkg-<namespace>-values secret with the new AVI certificate.

 

$ kubectl get secret tkg-pkg-<namespace>-values -n tkg-system
$ kubectl get secret avi-controller-ca -n tkg-system-networking
$ kubectl edit secret tkg-pkg-<namespace>-values -n tkg-system

The issue will be fixed in Incheon.1