NSX Edge Load Balancing fails when the VIP contains TCP port range from 10000-64000
Article ID: 321341
Updated On:
Products
VMware NSX for vSphere
Issue/Introduction
Symptoms:
In an environment where a Load Balancer Virtual Server IP contains TCP port range covering 10000-64000, you see these symptoms:
In an environment where a Load Balancer Virtual Server IP contains TCP port range covering 10000-64000, you see these symptoms:
- Load Balancing fails.
- In the Load Balancer logs, you see entries similar to:
loadbalancer[8626]: [default]: Connect() failed for backend pool-test: no free ports.
Environment
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x
Cause
Ports 10000-64000 are used by the Linux OS Kernel for several services such as HA proxy health check monitor, routing Service, HA rsync, etc. In addition, NSX Load Balancers use ports in this range to initiate connections to back-end pool members.
For example:
[root@NSX-edge-1-0 /var/db/vmware/vshield]# cat /proc/sys/net/ipv4/ip_local_port_range
10000 64000
When a VIP is configured for all port ranges from 10000-64000, it creates a network socket (IP Address/Port Bind) for the VIP IP with all the ports in this range.
For example:
[root@NSX-edge-4-0 ~]# netstat -anp | less
tcp 0 0 172.16.10.10:62343 0.0.0.0:* LISTEN 2311/haproxy
tcp 0 0 172.16.10.10:62311 0.0.0.0:* LISTEN 2311/haproxy
tcp 0 0 172.16.10.10:62279 0.0.0.0:* LISTEN 2311/haproxy
A total of 54000 sockets are created covering the full local port range.
For example:
[root@NSX-edge-4-0 ~]# netstat -anp | grep 172.16.10.10: | wc -l
54002
From the output below, the Pool Members have been marked down since there are no available ports in the Edge to initiate connections to the backend pool members hence the issue occurs.
For example:
NSX-edge-4-0> show service loadbalancer pool
-----------------------------------------------------------------------
Loadbalancer Pool Statistics:
POOL Web-Tier
| LB METHOD leastconn
| LB PROTOCOL L7
| Transparent disabled
| SESSION (cur, max, total) = (1, 2, 35)
| BYTES in = (4704), out = (8520)
+->POOL MEMBER: Web-Tier/Web-01_172.16.10.11, STATUS: DOWN
| | HEALTH MONITOR = BUILT-IN, default_https_monitor:L4CON
| | | LAST STATE CHANGE: 2018-02-28 17:30:10
| | | FAILURE DETAIL: Out of local source ports on the system
| | SESSION (cur, max, total) = (0, 1, 3)
| | BYTES in = (1488), out = (5112)
+->POOL MEMBER: Web-Tier/web-02a_172.16.10.12, STATUS: DOWN
| | HEALTH MONITOR = BUILT-IN, default_https_monitor:L4CON
| | | LAST STATE CHANGE: 2018-02-28 17:30:12
| | | FAILURE DETAIL: Out of local source ports on the system
| | SESSION (cur, max, total) = (1, 1, 3)
| | BYTES in = (2661), out = (3408)
For example:
[root@NSX-edge-1-0 /var/db/vmware/vshield]# cat /proc/sys/net/ipv4/ip_local_port_range
10000 64000
When a VIP is configured for all port ranges from 10000-64000, it creates a network socket (IP Address/Port Bind) for the VIP IP with all the ports in this range.
For example:
[root@NSX-edge-4-0 ~]# netstat -anp | less
tcp 0 0 172.16.10.10:62343 0.0.0.0:* LISTEN 2311/haproxy
tcp 0 0 172.16.10.10:62311 0.0.0.0:* LISTEN 2311/haproxy
tcp 0 0 172.16.10.10:62279 0.0.0.0:* LISTEN 2311/haproxy
A total of 54000 sockets are created covering the full local port range.
For example:
[root@NSX-edge-4-0 ~]# netstat -anp | grep 172.16.10.10: | wc -l
54002
From the output below, the Pool Members have been marked down since there are no available ports in the Edge to initiate connections to the backend pool members hence the issue occurs.
For example:
NSX-edge-4-0> show service loadbalancer pool
-----------------------------------------------------------------------
Loadbalancer Pool Statistics:
POOL Web-Tier
| LB METHOD leastconn
| LB PROTOCOL L7
| Transparent disabled
| SESSION (cur, max, total) = (1, 2, 35)
| BYTES in = (4704), out = (8520)
+->POOL MEMBER: Web-Tier/Web-01_172.16.10.11, STATUS: DOWN
| | HEALTH MONITOR = BUILT-IN, default_https_monitor:L4CON
| | | LAST STATE CHANGE: 2018-02-28 17:30:10
| | | FAILURE DETAIL: Out of local source ports on the system
| | SESSION (cur, max, total) = (0, 1, 3)
| | BYTES in = (1488), out = (5112)
+->POOL MEMBER: Web-Tier/web-02a_172.16.10.12, STATUS: DOWN
| | HEALTH MONITOR = BUILT-IN, default_https_monitor:L4CON
| | | LAST STATE CHANGE: 2018-02-28 17:30:12
| | | FAILURE DETAIL: Out of local source ports on the system
| | SESSION (cur, max, total) = (1, 1, 3)
| | BYTES in = (2661), out = (3408)
Resolution
VMware does not recommend configuring a Load Balancer VIP spanning the ip_local_port_range values on the edge from 10000 to 64000. Changing this value is not supported as this is not persistent if the NSX Edge is redeployed.
Additional Information
Impact/Risks:
Load balancer Monitoring would not work and will mark the pool members down. The NSX Edge will not be able to load balance the traffic since there are no available ports to initiate connections to the backend pool members.
Load balancer Monitoring would not work and will mark the pool members down. The NSX Edge will not be able to load balance the traffic since there are no available ports to initiate connections to the backend pool members.
Feedback
Yes
No
Powered by
