AKOO failing to reconcile when creating TKGm Workload clusters

Products

• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

• AKO Operator (AKOO) logs report an error syncing the ako-essential-role AVI role, e.g.:

2024-05-23T08:34:16.766Z ERROR controllers.AKODeploymentConfig failed to reconcile AKODeploymentConfig {"AKODeploymentConfig": {"name":"install-ako-for-all"}, "error": "Encountered an error on PUT request to URL https://<domain>/api/role/role-<UUID>: HTTP code: 500."

• AKO pods in newly created workload clusters are crashlooping due to AVI user credentials not found on cluster
• User for newly created workload cluster is not found in AVI controller

Environment

TKGm prior to v2.5.2

Cause

In TKGm versions prior to v2.5.2, there is a known issue in AKOO that causes it to fail to reconcile when any permissions have been added to the AVI role that it reconciles (ako-essential-role) outside of its own control loop. This can happen as a normal part of AVI controller upgrades; for example, AVI controller v22.1.1 added the “Auth Mapping Profile” permission to the ako-essential-role, triggering this known issue. When the role reconciliation fails, it then prevents AKOO from creating AVI users for new workload clusters.

TKGm v2.5.2 includes a bug fix to gracefully handle this scenario.

Resolution

The easiest way to remediate is generally to remove any permissions that have been added to the ako-essential-role out-of-band, so that the role definition remains in sync with what AKOO expects. As of yet, we have not seen any auto-added permissions that are actually required for AKO/AKOO.

This can be done by first exporting the ako-essential-role definition as JSON from AVI controller, and then doing a diff against the expected ako-essential-role permissions.
As of TKGm v2.5.0 and v2.5.1, the expected ako-essential-role definition are:

{
   "url": "https://<HOST>/api/role/role-<UUID>#ako-essential-role",
   "uuid": "role-<UUID>",
   "name": "ako-essential-role",
   "tenant_ref": "https://<HOST>}/api/tenant/admin#admin",
   "_last_modified": "<TIMESTAMP>",
   "allow_unlabelled_access": true,
   "privileges": [
       {
           "resource": "PERMISSION_VIRTUALSERVICE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_POOL",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_POOLGROUP",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_GSLBSERVICE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_GSLB",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_CLOUD",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_SERVICEENGINE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_SERVICEENGINEGROUP",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_NETWORK",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_GSLBGEODBPROFILE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_VRFCONTEXT",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_APPLICATIONPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_NETWORKPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_APPLICATIONPERSISTENCEPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_HEALTHMONITOR",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_ANALYTICSPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_IPAMDNSPROVIDERPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_CUSTOMIPAMDNSPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_TRAFFICCLONEPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_NATPOLICY",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_L4POLICYSET",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_HTTPPOLICYSET",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_NETWORKSECURITYPOLICY",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_AUTOSCALE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_DNSPOLICY",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_IPADDRGROUP",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_STRINGGROUP",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_SSLKEYANDCERTIFICATE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_SSLPROFILE",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_PKIPROFILE",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_AUTHPROFILE",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_PINGACCESSAGENT",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_CERTIFICATEMANAGEMENTPROFILE",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_HARDWARESECURITYMODULEGROUP",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_SSOPOLICY",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_VSDATASCRIPTSET",
           "type": "WRITE_ACCESS"
       },
       {
           "resource": "PERMISSION_PROTOCOLPARSER",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_WAFPOLICY",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_WAFPROFILE",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_WAFPOLICYPSMGROUP",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ERRORPAGEPROFILE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ERRORPAGEBODY",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ALERTCONFIG",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ACTIONGROUPCONFIG",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ALERTSYSLOGCONFIG",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ALERTEMAILCONFIG",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_SNMPTRAPPROFILE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_TRAFFIC_CAPTURE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_USER",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ROLE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_TENANT",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_SYSTEMCONFIGURATION",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_CONTROLLER",
           "type": "READ_ACCESS"
       },
       {
           "resource": "PERMISSION_REBOOT",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_UPGRADE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_TECHSUPPORT",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_INTERNAL",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_CONTROLLERSITE",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_USER_CREDENTIAL",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_ALERT",
           "type": "NO_ACCESS"
       },
       {
           "resource": "PERMISSION_IMAGE",
           "type": "NO_ACCESS"
       }
   ]
}