vSphere Supervisor Workload Cluster Load Balancer Service External IP Pending due to No available Free IPs

Products

• Tanzu Kubernetes Runtime
• VMware Avi Load Balancer

Issue/Introduction

Newly created load balancer services within a workload cluster are propagated to the Supervisor cluster successfully, but do not receive an external IP.

While connected to the workload cluster context with the affected new load balancer service:

• The load balancer service has <pending> state for its external IP:
  

  kubectl get svc -n <load balancer service namespace>

• Describing the load balancer service shows an error similar to the following, indicating that its external IP was not assigned:
  

  kubectl describe svc -n <load balancer namespace> <load balancer service name>
  
  "error syncing load balancer: failed to ensure load balancer: virtualmachineservice IP not found"

While connected to the Supervisor cluster context, the following symptoms are observed:

• The load balancer service created in the workload cluster is also present in the Supervisor cluster with <pending> state for the external IP:
  

  kubectl get virtualmachineservice -n <workload cluster namespace>

• Checking AVI-AKO pod logs show errors similar to the following:
  

  kubectl get pods -n vmware-system-ako
  kubectl logs -n vmware-system-ako <ako pod name>
  
  returned err {"code":0,"message":"map[error:No available free IPs!]", "Verb":"POST","Url":"<avi-manager-FQDN>:443//api/vs/vip","HttpStatusCode":400} with response null
  
  msg: Error in rest operation is not of type AviError, err: Error during POST: Encountered an error on POST request to URL <avi-manager-FQDN>:443//api/vsvip: HTTP code: 400: error from Avi: map[error:No available free IPs!]
  
  key: <user>/<domain-c>--<zone>-<load balancer service name from Supervisor cluster>, msg: no Free IP available, adding to slow retry queue

Environment

vSphere Supervisor

NSX-ALB

Cause

AVI cannot assign an external IP to the new load balancer service because the allocated IP pool or range in AVI does not have any available or free IP address to assign.

Resolution

You can resolve this issue by making IP addresses available for the new load balancer service.

• Work with your networking team to increase the IP pool or range in AVI.
  • This may include adjustments to the corresponding subnet or network profile.
    
    
• Free up unused or unnecessary IP addresses from the corresponding IP pool or range.
  • This can include cleaning up unused or unnecessary services in the vSphere Supervisor environment.

Additional Information

A YAML example of a simple load balancer service using nginx can be found below:

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vsphere-supervisor-services-and-standalone-components/latest/managing-vsphere-kuberenetes-service-clusters-and-workloads/deploying-workloads-on-tkg-service-clusters/pod-deployment-with-load-balancer-service.html