After changing AVI controller password, ako controller manager pod fails to start with error HTTP 401 "Invalid credentials"

Products

VMware vSphere with Tanzu

Issue/Introduction

vmware-system-ako-ako-controller-manager pod shows below error messages and fails to start -

vmware-system-ako                           vmware-system-ako-ako-controller-manager-69f65d6ddc-rhk2p         0/1     Error       1 (3s ago)       5s

2024-08-29T08:50:05.670Z	ERROR	cache/avi_ctrl_clients.go:72	AVI controller initialization failed
2024-08-29T08:50:05.670Z	FATAL	ako-main/main.go:243	Avi client not initialized
E0829 08:50:05.669727       1 avisession.go:383] response error: Encountered an error on POST request to URL https://192.168.10.50:443/login: HTTP code: 401; error from Avi: map[error:Invalid credentials]

Environment

VMware vSphere with Tanzu
NSX-T AVI Load Balancer

Cause

AKO is unable to initialise the connection with Avi Controller due to the authentication failure.
The HTTP 401 status code denotes that authentication has failed due to invalid credentials.

In this case specific case,
- the changed AVI credentials were not updated in the vCenter UI
- Ako Pods were not restarted

Resolution

Check that the credentials configured for AKO to access the Avi Controller are correct. These credentials are typically configured in the AKO ConfigMap or Secret.

Use the below command to get Ako secret and decode the password and username.

kubectl get secrets -n vmware-system-ako avi-secret -o yaml

---
apiVersion: v1
data:
  certificateAuthorityData: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMwVENDQWJtZ0F3SUJBZ0lVSnlBUHdNdjl6bWhlTFZaZ1lFcjBoQmc5eUJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dERVdNQlFHQTFVRUF3d05NVGt5TGpFMk9DNHhNQzQxTURBZUZ3MHlOREE0TURFd01UVTJNVFJhRncweQpOakE0TURFd01UVTJNVFJhTUJneEZqQVVCZ05WQkFNTURURTVNaTR4TmpndU1UQXVOVEF3Z2dFaU1BMEdDU3FHClNJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUM4OE96QW44dmVkN0RrU1hkSFAwR05INnBMNkt3Y0VOcHQKRjR3MDFEcDQyMDF4d2R0WmZqYnNjaVZhSFlXYUpJU25SUlZwb3l5RXBYeW1MaWpvUDdVT2x3c0RZSW5adXNzaAp4Vm83WWZmRG9nakRUaEFYNjVCdTBWN3p2bXF2S1FoTURsREw1dkh1SHFwRnRYS0FwL0t4ZjNTRVlld2h2S0o0Cnl3S0tJT0RQOUNIQ2FsbWdhUHBCT2NyOUZRR1IrY2dLdmtaU1h2ckNvaVRidjFNZWtmK1NKUXdFdXBzMTkvMkgKclRjVFo1VDNXZm51SEx1aUhyK0wwZitiZ2hPZ1ZrdjNtY1UzZmhrZVFySDdaVHlwTGtrak84RWVka09pdHZWdAo5T09WUkR2eW10cjBNNlRlWGxRS0g4cnVla29BV1NySWhMaGF5YmJvMWdPdjVzSW9FZm9iQWdNQkFBR2pFekFSCk1BOEdBMVVkRVFRSU1BYUhCTUNvQ2pJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLM1FjdFlqQ2svRnlRYTMKalhYTHRsYm5TeFBvTWZub0s1Wm5OamhUa3J2UlIzb3Y2THM1cVNOSTBqdUJ1dmZnS1FTTUEwbGtPRHFsVXdlRApZaE1Ec2h3elZxVjd4bmV4aXFENHNiQjRnQ0tnZUtFK3FvS0lldVZOL01zRWhVSFRYMWF2KzM3SjdqME9IRmJRCmx2K2ZsWW9VUHJYbFgxdGdQSm94NHMzOHdKVGwzZDBUZVl2ZG1ESFVXTGxvdzRjazNDWVNUUE1IV2RpZXB3ZGUKYUFUSmU1YmZCNXgwYWdUVitRU1BHNVp2SkxPeno1Mzl6Z1FldG8vVTUzM2hPNDlDV2xTdGJhU0ZhN2VpVFkyQQpqVTNSSDA5L3RDcHpzL0F6Z3UvQmVIMnNpUHlxbTdKNWpzbldiNndYa1hXQUNiam1qTFZSbUhmWjJoNTNKNFFIClY5WXNvMVU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  password: QWRtaW5AMTIzNA==
  username: YWRtaW4=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"avi-secret","namespace":"vmware-system-ako"}}
  creationTimestamp: "2024-08-01T02:12:27Z"
  name: avi-secret
  namespace: vmware-system-ako
  ownerReferences:
  - apiVersion: netoperator.vmware.com/v1alpha1
    kind: LoadBalancerConfig
    name: nsx-alb
    uid: 808c5841-7878-4924-8f96-50534dda8afb
  resourceVersion: "23657130"
  uid: fdb69eec-c3b9-4284-ba29-5810de719810
type: Opaque
---

if there is a mismatch with the ako secret and the avi controller, please below step.

To update the new credentials for the secret in vcenter UI.

workload management > supervisor cluster > configure> network

under loadbalancer, update the new password

3. Delete the AKO pod and the new pod get recreated.

kubectl delete pod -n <ako-podns> <ako-podname>

By following these steps, should be able to identify and resolve the issue with Avi Controller initialization in AKO deployment.

Additional Information

To create a new user/update a new password to avi controller.

- create a new user

Administration -> users -> create

- To update a new password for current user

as seen in below snippet, navigate to account and provide the new password.