New Supervisor deployment in VCF 9.0 gets stuck in "Configuring" on "Configured Load Balancer fronting the Kubernetes API Server"

Products

• VMware Cloud Foundation

Issue/Introduction

• In a vCenter Server with more than 1 Supervisor instance using vSphere Foundation Load Balancer, the deployment of another Supervisor instance gets stuck in "Configuring" on "Configured Load Balancer fronting the Kubernetes API Server Timed out waiting for LB service update. This operation is part of the cluster enablement and will be retried".
  
  
• The /var/log/pods/vmware-system-netop_vmware-system-netop-controller-manager-* logs of the supervisor cluster look like below.
  
  controllers/LoadBalancerConfig/<flb id> "msg"="unable to reconcile FoundationLoadBalancerConfig" "error"="[400 Bad Request: {"error_type":"ALREADY_EXISTS","messages:[{args:[],default_message:locmsg.com.vmware.vcenter.flb.AlreadyExists,id:com.vmware.vcenter.flb.AlreadyExists,params:{id:{s:<flb node id>},object:{s:foundation load balancer node}}}]},
failed to reconcile FoundationLoadBalancerConfigStatus: FoundationLoadBalancerConfig is missing the 'vmware.com/foundation-load-balancer-id' annotation]" "name"="<flb id>

• As a result, the Foundation LoadBalancer VMs for a subsequent Supervisor instance may get stuck. On top of this, the Supervisor Control Plane VMs may never obtain a Virtual IP. 

Environment

VCF 9.0.0.0

Cause

This is caused due to the Foundation Load Balancer instance naming conflict. In environments with many vSphere Supervisor instances that use VMware NSX Advanced Load Balancer (formerly Avi Networks), if you try to deploy load balancers with the same name, the deployment fails and Supervisor Control Plane VMs' cannot get a virtual IP. 

Resolution

This issue is expected to be addressed in one of the future VCF releases.

 

Workaround:

Provide unique names to load balancers. Since load balancer names are not editable, in case the issue reoccurs, the Supervisor instance needs to be recreated.