IP addresses of Kubernetes load balancer deployed with static IP (loadBalancerIP) are not marked as allocated in NSX-T IP pools

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Kubernetes load balancers are deployed with static IP using the loadBalancerIP setting.

Example of yaml file:
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  loadBalancerIP: 10.10.10.42
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  type: LoadBalancer

• The IP address assigned to the load balancer is part of the namespace NSX-T IP pool and the IP is not marked as allocated in the NSX-T IP pool
• NCP version in used at the time the load balancer is deployed is pre-NCP 2.5.1.

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Cause

Pre-NCP 2.5.1, NCP did not track the allocation of static load balancer IP addresses. As a result, NCP may allocate the IP address to a different services causing IP conflicts.

Resolution

This issue is resolved in NCP 2.5.1.
Note: if IP conflicts already exist (i.e. the load balancer IP as been associated for a different service pre-NCP 2.5.1) refer to the Workaround section below.

Workaround:
Upgrading to NCP 2.5.1 does not fix the issue if IP conflicts already exist (i.e. the load balancer IP as been associated for a different service pre-NCP 2.5.1).
As a workaround, manual operation is required to clear the IP conflicts and you need to either:

• Release the IP from other services.
• Change the load balancer IP (loadbalancerIP) to a free IP.

As a workaround, if upgrading to NCP 2.5.1 is not an option, you can manually allocate the load balancer IP addresses in the NSX-T IP pool to ensure they are not assigned for any other services:

1. Find the IP pool id using REST API:
GET /api/v1/pools/ip-pools
Example of output:
{
  "results" : [ {
    "pool_usage" : {
      "total_ids" : 248,
      "allocated_ids" : 1,
      "free_ids" : 247
    },
    "subnets" : [ {
      "cidr" : "169.254.17.0/24",
      "dns_nameservers" : [ ],
      "allocation_ranges" : [ {
        "start" : "169.254.17.3",
        "end" : "169.254.17.250"
      } ]
    } ],
    "resource_type" : "IpPool",
    "id" : "########-####-####-####-##########d5", <<---- IP Pool id
    "display_name" : "SI_Destination_IP_Pool",
    "description" : "ServiceInsertion_IP_Pool_for_NS_insertion_at_T0/T1",
    "_create_user" : "system",
    "_create_time" : 1579798571269,
    "_last_modified_user" : "system",
    "_last_modified_time" : 1579798571269,
    "_system_owned" : true,
    "_protection" : "NOT_PROTECTED",
    "_revision" : 0
  }

2. Mark the IP address as allocated in the IP pool using REST API:
POST /api/v1/pools/ip-pools/{IP_POOL_ID}?action=ALLOCATE
Body:
{
  "allocation_id":"{IP_ADDRESS}"
}

3. Verify the allocation using REST API:
GET /api/v1/pools/ip-pools/{IP_POOL_ID}/allocations

Note: if you delete the Kubernetes load balancer, the manual allocations above would need to be manually deleted using the following REST API:
POST /api/v1/pools/ip-pools/{IP_POOL_ID}?action=RELEASE
Body:
{
  "allocation_id":"{IP_ADDRESS}"
}