Error: tls: failed to verify certificate: x509: certificate signed by unknown authority

Products

• Tanzu Kubernetes Runtime
• VMware Avi Load Balancer

Issue/Introduction

• AVI Controller Certificate not updated after updating avi-controller-ca secret in TKG
  
  
• The AVI Controller CA has been updated by patching the avi-controller-ca as per the docs on how to modify the Avi Controller Certificates
  
  
• The ako and ako-operator show that they can't connect to the AVI Controller due to an unknown certificate authority.
  avisession.go:666] Client error for URI: login. Error: Post "https://<AVI Controller>/login": tls: failed to verify certificate: x509: certificate signed by unknown authority

Environment

TGKm: 2.5.4

ALB: 22.1.3

Cause

During AVI CA rotation, the new CA was patched into the cluster secrets, but AKO continued using a stale or incomplete trust bundle due to secret name/namespace mismatch 

Resolution

1. Update the tkg-pkg-system-values with the new certificate.
   kubectl get secret -n tkg-system tkg-pkg-tkg-system-values -o jsonpath='{.data.tkgpackagevalues\.yaml}' | base64 -d > tkgpackagevalues.yaml
   
   Note: In the TKGm 2.4. x release, the secret name is "tkg-pkg-tkg-system-addon" in the tkg-system namespace. 
   
   
2. Update CA in tkgpackagevalues.yaml with the new avi encoded certs
   
   
3. Encode the updated tkgpackagevalues.yaml
   cat tkgpackagevalues.yaml | base64 -w 0
   
   
4. Update  tkg-pkg-tkg-system-addon secret with encoded contents of tkgpackagevalues.yaml
   kubectl  edit secret -n tkg-system tkg-pkg-tkg-system-values