TCP Half Open connections exceeding the Maximum causes issues with TCP/UDP communication

Products

• VMware NSX

Issue/Introduction

Known issue regarding TCP Half Open connections exceeding maximum
.

Symptoms:

This issue is prevalent in NSX-T versions below 2.5.2, and the symptoms may manifest in different forms. Here are some examples.

All the TCP/UDP traffic gets dropped at Tier-0/Tier-1
or
NSX-T Load balancer stops processing all the traffic
or
The HA state on the Edge shows as Unknown.
ICMP traffic works fine
Issue is seen only after traffic burst

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

The value 4294967295 equates to -1. The command stated above gives the dump of the current state, but it is not possible to predict when the system will get into an error state as the "TCP Half Opened Active/Max" number does NOT increment over time and reach the MAX(4294967295) value, Instead, this error state happens sporadically when the connection counter drops below zero.

The NSX-T versions above 2.5.2 are better equipped to handles the decrement of the "TCP Half Opened Active/Max" below zero by avoiding it and this issue has not been seen in NSX-T 2.5.2 or newer.

Resolution

Upgrade the NSX-T to 2.5.2 and above.
NSX-T 3.2.0 and above has some additional enhancements.

Workaround:
- On the live setup you see "TCP Half Opened" is at the Max value of 4294967295 for some or all the interfaces.

root@edge-x:~# su admin -c get firewall interfaces | grep Interface
Wed Feb 23 xxxx UTC xx:xx:xx:xx
Interface : xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx

root@edge-1:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl fw/get_sessioncount xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
{"TCP Half Opened":4294967295,"UDP Active":0,"ICMP Active":0,"Other Active":0,"TCP Half Opened MAX":1000000,"UDP Active MAX":100000,"ICMP Active MAX":10000,"Other Active MAX":10000}


NOTE: In newer version the command "edge-appctl -t /var/run/vmware/edge/dpd.ctl fw/get_sessioncount xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" may not work, instead make use of the command "edge-appctl -t /var/run/vmware/edge/dpd.ctl fw/show fw-lr-connections | python -mjson.tool"

To clear the "TCP Half Opened Active/Max" entries

 - Reboot the Edge
or
 - Restart the Edge Data-plane
 restart service dataplane

Additional Information

Impact/Risks:
Traffic interruption for TCP/UDP traffic traversing through the Edge.