AKO Controller pod in CrashLoopBackOff state with error "Configured Load Balancer fronting the Kubernetes API Server Timed out waiting for LB service update.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vSphere Kubernetes Supervisor Cluster deployment using vSphere Distributed Switch networking and AVI as the load balancer fails to complete with the following error on vCenter Server UI "Configured Load Balancer fronting the Kubernetes API Server Timed out waiting for LB service update. This operation is part of the cluster enablement and will be retried."
  
  
  
• wcpsvc.log on vCenter Server reports the following error indicating a timeout while waiting for the Load Balancer Virtual IP: 
  Log location: /var/log/vmware/wcp/wcpsvc.log
  YYYY-MM-DDTHH:MM:SS info wcp reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch clusters: failed to list clusters: failed to list clusters: the server could not find the requested resource
YYYY-MM-DDTHH:MM:SS error wcp [apiserver/manager.go:158] [opID=#####-#####-#####-#####-#####] timed out waiting for kube-apiserver-lb-svc service update. Err: context deadline exceeded
YYYY-MM-DDTHH:MM:SS error wcp [kubelifecycle/controller.go:1830] [opID=#####-#####-#####-#####-#####] An error occurred fetching the virtual IP: Timed out waiting for LB service update. This operation is part of the cluster enablement and will be retried.. Retrying


• The ako-controller pod in the vmware-system-ako namespace is stuck in a CrashLoopBackOff state when checked on Supervisor cluster: 
  kubectl get pods -A | grep -i vmware-system-ako-ako-controller-manager
vmware-system-ako                           vmware-system-ako-ako-controller-manager-6d55d944f6-7tqv4         0/1     CrashLoopBackOff 


• The logs for the ako-controller pod reveal a segmentation violation and a panic caused by an incorrect cloud type (CLOUD_NSXT):
  kubectl logs <ako-controller pod name> -n vmware-system-ako
YYYY-MM-DDTHH:MM:SS INFO utils/avi_rest_utils.go:104 Setting the client version to the current controller version 22.1.2
YYYY-MM-DDTHH:MM:SS INFO cache/controller_obj_cache.go:2343 Avi cluster state is CLUSTER_UP_HA_ACTIVE
YYYY-MM-DDTHH:MM:SS INFO cache/controller_obj_cache.go:2910 Setting cloud vType: CLOUD_NSXT
YYYY-MM-DDTHH:MM:SS INFO cache/controller_obj_cache.go:2913 Setting cloud uuid: cloud-#####-#####-#####-#####-#####
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x17cc141]

Environment

VMware vSphere Kubernetes Service
VMware vSphere Supervisor Cluster

Cause

• This issue occurs when there is a mismatch between the underlying vSphere networking architecture and the Cloud Type configured within the Avi Load Balancer (NSX ALB) Controller.
• In this scenario, the Supervisor Cluster is being deployed using vSphere Distributed Switch networking. However, the ako-controller logs show it is pulling a configuration for an NSX-T Cloud (vType: CLOUD_NSXT). Because AKO expects an NSX-T backend but is running on VDS, it encounters a nil pointer dereference when checking for usable network labels, causing the controller to panic, crash, and fail to assign the necessary Virtual IP for the Kubernetes API Server.

Resolution

To resolve this issue, the Cloud type within the Avi Load Balancer Controller needs to be configured with vCenter Cloud to match the VDS networking environment.

1. Log in to the Avi Load Balancer Controller UI.
2. Delete the incorrect NSX-T Cloud configuration.
3. Recreate the Cloud configuration, ensuring you specifically select vCenter as the Cloud Type.
4. Ensure all IPAM, DNS, and Service Engine Group settings are mapped correctly to the new vCenter Cloud.

For step-by-step instructions on properly configuring the vCenter Cloud type for AKO with VDS, refer to the following documentation: Deploying AKO via Helm into VKS on vSphere Supervisor with VDS Networking

Once the Avi Controller is correctly configured with a vCenter Cloud, the ako-controller pod will initialize successfully, and the Supervisor cluster deployment will automatically resume and complete the Load Balancer configuration step.

Additional Information

Requirements for Supervisor Deployment with Avi Load Balancer and VDS Networking