Unexpected IP_POOL_EXHAUSTED when updating loadBalancerIP in load balancer service

Products

VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Create a service of type loadbalancer using IP pool ID:

Following https://techdocs.broadcom.com/us/en/vmware-cis/nsx/event-catalog/4-2/nsx-container-plugin-for-kubernetes-and-tanzu-application-service/load-balancing/service-of-type-loadbalancer.html

Two components are essential:

forceUsePool: "true"

loadBalancerIP: d4600c49-xxxx-xxxx-xxxx-956bfd41af88

Where the <ID> is the floating IP pool ID created follwoing the above page.

apiVersion: v1
kind: Service
metadata:
  name: my-test-0
  namespace: default
  labels:
    app: my-test-0
  annotations:
    forceUsePool: "true"
spec:
  type: LoadBalancer
  loadBalancerIP: d4600c49-xxxx-xxxx-xxxx-956bfd41af88
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilyPolicy: SingleStack
  ports:
    - name: http-9992
      port: 9992
      targetPort: 10101
      protocol: TCP
  selector:
    app: my-test-0
  sessionAffinity: None

This issue happenes only when UPDATE operation is applied to the loadBalancerIP i.e if the loadbalancer service is updated with another ID from different IP pool

where only one IP is available, instead of getting the free IP an error is recorded with the following errors visible on the service object:

kubectl describe svc my-test-0
Name:                     my-test-0
Namespace:                default
Labels:                   app=my-test-0
Annotations:              forceUsePool: true
                          ncp/error.loadbalancer: IP_POOL_EXHAUSTED
                          ncp/error.loadbalancer.unrealized_ports: 9992
                          ncp/nsxerror: services-68326433-xxxx-xxxx-xxxx-1934f5a6d78f
Selector:                 app=my-test-0
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.xxx.xxx.203
IPs:                      10.xxx.xxx.203
Desired LoadBalancer IP:  d4600c49-xxxx-xxxx-xxxx-956bfd41af88
Port:                     http-9992  9992/TCP
TargetPort:               10101/TCP
NodePort:                 http-9992  30611/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:                   <none>

Also an error is observed in the NCP logs on the master nodes:

2025-09-09T15:39:04.806Z d1c8c01a-xxxx-xxxx-xxxx-def65f0c2920 NSX 112444 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="INFO"] nsx_ujo.ncp.inventory Inventory send out update [('UPDATE', {'network_status': 'UNHEALTHY', 'network_errors': [{'error_message': "Failed to allocate ip from ip pool: Unexpected error from backend manager (['nsxmgrx-tanzu.local']) for POST api/v1/pools/ip-pools/d4600c49-xxxx-xxxx-xxxx-956bfd41af88?action=ALLOCATE: Insufficient free IP addresses to allocate from the pool.", 'error_code': 'NCP00015', 'spec': '{"origin": "68326433-xxxx-xxxx-xxxx-1934f5a6d78f"}'}], 'external_id': '68326433-xxxx-xxxx-xxxx-1934f5a6d78f', 'resource_type': 'ContainerApplication'}), ('UPDATE', {'network_status': 'UNHEALTHY', 'network_errors': [{'error_message': "Failed to allocate ip from ip pool: Unexpected error from backend manager (['nsxmgrx-tanzu.local']) for POST api/v1/pools/ip-pools/d4600c49-xxxx-xxxx-xxxx-956bfd41af88?action=ALLOCATE: Insufficient free IP addresses to allocate from the pool.", 'error_code': 'NCP00015', 'spec': '{"origin": "68326433-xxxx-xxxx-xxxx-1934f5a6d78f"}'}], 'external_id': '8154ab21-xxxx-xxxx-xxxx-eb1da385f8f4', 'resource_type': 'ContainerCluster'})]

Also from NSX manager errors of similar type can be found:

[nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP05025"] nsx_ujo.ncp.k8s.service_lb_controller Failed to process L4 Lb service ('default', 'protect-me-with-       ipreservation-c'): LbLayer4Service.update_lb_virtual_server_attributes() got an unexpected keyword argument 'tenant'

Environment

TKGi 1.21

TKGi 1.22

Cause

Problem have been found and addressed in NCP 4.2.2 and the above two TKGi versions will be patched with the updated version of NCP in the comming releas cycle:

NCP bump to 4.2.2 will be provided in:

1.21.2 Patch

1.22.3 Patch