AKO operator unable to communicate with AVL controller due to AKO pod being stuck in CrashLookBack

Products

• VMware Telco Cloud Automation

Issue/Introduction

AKO operator added as an add-on to the TCA management cluster
AKO tasks not seen in AVI logs
AKO pod stuck in CrashLookBackoff
AKO Events showing OOMKilled,  Exit Code: 137

Environment

2.3

Cause

The controller version is 21.1.6, but Ako logs show the controller version set to 21.1.2.
In the config map, no ControllerVersion field value is provided [If no controller version was mentioned in the value.yaml file], then AKO fetches data from API.
AKO restart is happening due to a version mismatch.

 

When AKO POD restarted, it would have taken memory allocated to POD. On each bootup, AKO initializes a few objects, and then AKO goes into a restart loop. This is happening so fast that the GO garbage collector couldn’t have had an opportunity to claim back memory, and POD went out of memory.

Resolution

Manually pause pkgi and change the AKO memory limit to 4000Mi (from 400Mi). 

Verify with 'kubectl top pod -n avi-system ako-0' shows memory usage at 313Mi