Manually patch ako operator addon secret with new AVI certificate

Products

• Tanzu Kubernetes Grid

Issue/Introduction

• Try to upgrade TKG to 2.1.0(and above)
  avisession.go:666 Client error for URI: login. Error: Post “https://<avi ip|url>/login”: x509: certificate signed by unknown authority
  
  
• Rotated the AVI certificate

Environment

TKGm: 2.5.x

Cause

This issue happens because customers rotated the AVI certificate before the upgrade, but this new value won’t be updated in the ako operator addon secret/tkg-pkg secret. When upgrading to a 2.1 (and above) cluster, the ako operator will be reconciled by tkg-pkg, and the AVI certificate value is set to the value in the tkg-pkg secret. Upgrade will fail because AVI still uses the old AVI certificate.

Resolution

Legacy cluster:

1. Before the upgrade, check the AVI certificate in <cluster-name>-ako-operator-addon and the AVI certificate in avi-controller-ca. If two values are different, manually patch  <cluster-name>-ako-operator-addon secret with the new AVI certificate.
   
   kubectl get secret <cluster-name>-ako-operator-addon -n tkg-system

kubectl get secret avi-controller-ca -n tkg-system-networking

kubectl edit secret <cluster-name>-ako-operator-addon -n tkg-system

kubectl edit secret avi-controller-ca -n tkg-system-networking

 

Classy cluster

Before the upgrade, check the AVI certificate in tkg-pkg-<namespace>-values and in avi-controller-ca. If the two values differ, manually patch the tkg-pkg-<namespace>-values secret with the new AVI certificate.

1. Backup
   
   kubectl get secret tkg-pkg-tkg-system-values -n tkg-system -o yaml > tkg-values-backup.yaml
   
   
2. Pull the new cert from the controller. Swap in your actual controller IP:
   
openssl s_client -connect #.#.#.#:443 -showcerts </dev/null 2>/dev/null | openssl x509 -outform PEM > avi-controller.crt
   
   
3. Then encode it – you'll need this variable for the rest of the steps, so keep the same terminal open:
   
   NEW_CA=$(cat avi-controller.crt | base64 -w 0)
   
   
4. Update the management secret by extracting the current config, patching the cert value in, then pushing it back up:
   
   kubectl get secret tkg-pkg-tkg-system-values -n tkg-system -o jsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d > tkg-internal-config.yaml

sed -i "s/AVI_CA_DATA_B64: .*/AVI_CA_DATA_B64: $NEW_CA/" tkg-internal-config.yaml
sed -i "s/avi_ca_data_b64: .*/avi_ca_data_b64: $NEW_CA/" tkg-internal-config.yaml

kubectl patch secret tkg-pkg-tkg-system-values -n tkg-system --type='json' -p="[{\"op\": \"replace\", \"path\": \"/data/tkgpackagevalues.yaml\", \"value\":\"$(cat tkg-internal-config.yaml | base64 -w 0)\"}]"
   
   
5. Update the networking secret. The AKO pod needs this updated separately or it won't trust the controller:
   
   kubectl patch secret avi-controller-ca -n tkg-system-networking --type='json' -p="[{\"op\": \"replace\", \"path\": \"/data/certificate\", \"value\":\"$NEW_CA\"}]"
   
   
6. Verify
   kubectl get secret tkg-pkg-tkg-system-values -n tkg-system -o jsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d | grep "AVI_CA_DATA_B64"
   
   
7. Should show your new cert string. If the AKO pod doesn't come good on its own, bounce it:
   
   kubectl rollout restart deployment/ako -n avi-system