Resize Load Balancer managed by NSX-T and NCP in vSphere Supervisor

Products

VMware vSphere Kubernetes Service

Issue/Introduction

At this time, the size of an existing NSX-T load balancer cannot be changed in the vSphere Supervisor, formerly vSphere with Tanzu, product.

The new load balancer service does not receive an external IP address, showing as <pending>:

kubectl get svc -n <namespace> | grep pending

NAMESPACE       NAME                           TYPE            INTERNAL IP      EXTERNAL IP
<namespace>     <loadbalancer service name>    LoadBalancer    <internal IP>    <pending>

When describing the new load balancer service with <pending>, the error messages similar to the following are present, where the values encased in brackets <> will vary by environment:
- In the below error, the <LB limit> is determined by the size of the load balancer associated with the namespace:
```
kubectl describe svc -n <namespace> <loadbalancer service name>

nsx-container-ncp  LB Service <load balancer> limit exceeded: Unable to attach new resource <new member> to lbs <load balancer>: LBS exceeded limit of <LB limit>.
```
- This service will also have the following annotation:
```
Annotations:     ncp/error.loadbalancer: LBS_LIMIT_EXCEEDED
```

While connected to the Supervisor cluster context, the NSX-NCP pod log may show errors similar to the following:

kubectl get pods -n nsx-ncp

kubectl logs -n nsx-ncp <nsx ncp pod> -c nsx-ncp

The maximum size of pool members for <load balancer SIZE> load balancer service form factor is <load balancer size limit>, current size of pool members is <greater than or equal to the load balancer size limit>"

Environment

vSphere Supervisor

NSX-T 4.X

NCP (NSX-T Container Plugin) 4.X

Cause

Load Balancer autoscaling works only against number of virtual servers, not the pool member count.
This is expected behavior by design as relocating the services can require downtime.
The default load balancer size is SMALL, which has a limit of 300 pool members by default in NSX-T. However, this is overridden by NSX-NCP in vSphere Supervisor using NSX-T.

In a NSX-T load-balancer, pool-members are created to distribute the traffic between them. Every pool member is an object containing a unique pool-member-IP+Port combination.

Resolution

vSphere Supervisor currently does not support changing the size of an existing load balancer. The only solutions are the following:

Modify the existing workloads for the namespace and/or cluster to reduce the members for the affected load balancer.
Create a new workload cluster with a larger load balancer size and migrate workloads to that new cluster.
- NSX-NCP overrides the default load balancer pool member limit in vSphere Supervisor using NSX-T.
  - Small Pool Member Limit: 2,000
  - Medium Pool Member Limit: 2,000
  - Large Pool Member Limit: 6,000
Create multiple namespaces with workload clusters to disperse the migrated workloads across multiple load balancers of sufficient size.

A namespace can be created with a larger load balancer through using Override Cluster Network Settings:

Override Workload Network Settings for a vSphere Namespace

Additional Information

WARNING: It is not supported to edit the nsx-ncp-config configmap configuration in a vSphere Supervisor environment.

Changes made to the nsx-ncp-config configmap will be reverted on Supervisor control plane node recreation, such as a Supervisor cluster upgrade.

This will inevitably cause application downtime and errors on the affected load balancers.