K8S service creation stuck with waiting for service IP with error "Exceed maximum number of pools on the Load balancer service"

Products

• VMware
• VMware NSX Networking

Issue/Introduction

Symptoms:

• When you create a K8S service, its pending at waiting for service IP.

• In the ncp logs, you see that server pool limit is exhausted on the cluster load balancer.

• In the /var/vcap/sys/log/ncp/ncp.stdout.log on master node, you see the entries similar to:
  response body {u'error_code': 23500, u'related_errors': [{u'error_code': 23704, u'error_message': u'Exceed maximum number of pools on the Load balancer service. The maximum size of pools for SMALL load balancerservice form factor is 20, current size of pools is 21.
  1 2019-08-08T14:23:03.213Z 543c4597-5883-4078-94f8-b9c69fd144e6 NSX 17660 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP00050"] nsx_ujo.ncp.nsx.manager.base_k8s_nsxapi Failed to UPDATE virtual server

• When you check the LB server pool count from edge vm, you see that pool limit is already reached.
  get load-balancer <load-balancer-ID> pools

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x
VMware PKS 1.x

Cause

This issue occurs when the server pool limit is exhausted on the PKS cluster load balancer. In VMware NSX-T 2.3.1 supports only 20 server pools per small load balancer. For more information, see Scaling Load Balancer Resources section in VMware NSX-T 2.3.1 Documentation .

Auto scaling for PKS cluster LB was introduced in NCP and it creates a new load balancer if the current LB reaches the Virtual Server (VS) limit, however we don’t have any auto scaling if the server pool count reaches the limit.

Resolution

To resolve this issue, upgrade to VMware NSX-T 2.4 to increase the Small LB server pool limit to 60.

For more information, see Scaling Load Balancer Resources section in VMware NSX-T 2.4 Documentation .