vSphere Supervisor Cluster Deployment Stuck using Avi Load Balance- Configured Load Balancer fronting the kubernetes API Server. Server closed the connection while watching LB service. This operation is part of the cluster enablement and will be retried.

Products

• Tanzu Kubernetes Runtime
• vSphere with Tanzu
• VMware vSphere Kubernetes Service
• VMware vSphere 7.0 with Tanzu
• VMware Avi Load Balancer

Issue/Introduction

Initial vSphere Supervisor cluster deployment is stuck with the following Warning:

"Configured Load Balancer fronting the kubernetes API Server.  Server closed the connection while watching LB service. This operation is part of the cluster enablement and will be retried."

 

While connected to the Supervisor cluster context, the following symptoms are observed:

• All csi-controller pods are in CrashLoopBackOff state with similar error messages to the following:

  • kubectl get pods -n vmware-system-csi
    
    

  • The following command specifically checks the container "vsphere-csi-controller", however, the below error message is present in the "vsphere-syncer' container:
    • If AvailabilityZone(s) were properly set up in the environment, this may indicate an issue with the load balancer communicating to the vCenter FQDN.

    • kubectl logs -n vmware-system-csi <vsphere-csi-controller pod> -c vsphere-csi-controller
      
      "msg":"failed to get clusterComputeResourceMoIds. err: could not find any AvailabilityZone"
      
      

• Because the csi-controller pods are failing, the vmop-controller-manager and psp-operator-mgr system pods will also be in CrashLoopBackOff due to their dependency on csi objects.
  • vmop-controller-manager logs will show an error message similar to the following:

    • problem creating controller manager" err="failed to add resources to the manager: failed to initialize Volume controller: no matches for kind \"CnsNodeVmAttachment\" in version \"cns.vmware.com/v1alpha1\"" logger="entrypoint"
      
      

  • psp-operator-mgr logs error messages will be similar to the below:

    • "msg":"Error starting a watch for StoragePools","err":"the server could not find the requested resource"
      
      "msg":"Shutting down due to error","error":"the server could not find the requested resource"
      
      

• The load balancer pod in the Supervisor cluster is in CrashLoopBackOff state with similar error messages to the following:
  • VMware Avi Advanced Load Balancer (Avi-ALB) uses the AVI-AKO pod.

    • kubectl get pods -A | egrep "ako|ncp|lbapi"
      
      

    • The AVI-AKO pod error message references a networking object which contains spaces in its name:

      • kubectl logs -n vmware-system-ako <vmware-system-ako-ako-controller-manager pod>
        
        Unable to fetch collection data from uri /api/network/?include_name&page_size=100&name.in=<networking object name> Encountered an error on GET request to URL https://<vCenterFQDN>:443//api/network/?include_name&page_size=100&name.in=<networking object name>: HTTP code: 400; error from Avi: <nil>

 

From the vSphere web client, the above referenced networking object  contains spaces in its name.

Environment

vSphere 7.0 with Tanzu

vSphere 8.0 with Tanzu

VMware Avi Advanced Load Balancer (Avi-ALB)

Cause

VMware Avi Advanced Load Balancer (Avi-ALB) for vSphere Supervisor does not support using spaces in the name of networking components.

Avi-ALB will fail to perform GET requests against the vCenter FQDN to fetch the networking object due to the spaces in name of the networking object.

Resolution

Use networking components that do not contain spaces in the name.