Supervisor Deployment Fails with AKO Pods in CrashLoopBackOff due to AVI Cloud Configuration Mismatch

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During the deployment of a Supervisor cluster in VMware Cloud Foundation (VCF) 9.1, the process fails because the Avi Kubernetes Operator (AKO) pods (vmware-system-ako-ako-controller-manager) enter a CrashLoopBackOff state.

This prevents the assignment of Virtual IPs (VIPs) for the Supervisor cluster.

Log analysis of the infra container within the AKO pod reveals the following fatal error:

FATAL infra-main/main.go:141 Failed to derive cloud, err: cloud not found matching transport zone Default-Cloud

Environment

VCF - 9.1
vCenter - 9.1
VKS - 3.6.1-embedded + v1.35
AVI Load Balancer - 32.1.1
AKO - 3.0.1 (ob-25219785)

Cause

The issue is caused by a configuration mismatch and an architectural limitation within the AVI environment:

1. Default Cloud Conflict - AKO incorrectly attempts to default to "Default-Cloud" when a specific VPC-enabled cloud is required.
2. Architectural Limitation - The AVI cluster is restricted to a "one VPC cloud per AVI cluster" limitation. In this scenario, the presence of multiple clouds (e.g., existing NSX Clouds and a "No Orchestrator" cloud) blocks the enablement of the required VPC Mode on the desired cloud.
   
   
3. Mapping Requirement - VPC deployments require a strict 1:1 mapping between the Workload Domain (WLD), NSX Manager, and AVI Controller to function correctly in this architecture.

Resolution

To resolve the deployment failure, align the environment with the supported VPC architecture:

1. Remove Conflicting Clouds - Log into the AVI Controller and delete any failed or unnecessary clouds, including the "No Orchestrator" (Default Cloud) and any redundant NSX Clouds that are not part of the intended VPC deployment.
   
   
2. Adhere to 1:1 Architecture by ensuring that each Workload Domain (WLD) utilizing VPC support is assigned a dedicated NSX Manager and a dedicated AVI Controller cluster.
    - Note: While NSX Managers can be shared across WLDs in certain 1:N deployments, the VPC requirements for Supervisor often necessitate the dedicated 1:1 model.
   
   
3. Redeploy Supervisor - Once the AVI cloud configuration is cleaned and the architecture is aligned, restart the Supervisor deployment.

Additional Information

Further information can be found here.