TCP Health Monitor Failures Caused by Source Port Exhaustion and Connection Collisions

Products

• VMware Avi Load Balancer

Issue/Introduction

Setup:

                        

• TCP health monitors intermittently fail with connection reset errors.

• Virtual Service (VS) traffic experiences intermittent disruption because TCP health monitors intermittently transition to a DOWN state. Packet captures confirm that these monitor failures are triggered by TCP connection resets (RST) occurring during the TCP handshake phase.

• As a result, affected pool members are marked unhealthy, which leads to traffic impact at the Virtual Service level.

Cause

• The backend device (F5) reports that it is handling more than 65,000 concurrent TCP connections originating from the same source IP. This condition leads to source port exhaustion, where the available ephemeral source port range is fully utilized.
• When a new TCP connection attempt is initiated while an existing or half-closed connection is still occupying the same 4-tuple (source IP, source port, destination IP, destination port), the F5 detects the collision and responds by sending a TCP RST before the handshake can complete.
• This behavior is expected from the TCP stack under port exhaustion conditions and explains the observed resets impacting TCP health monitors.

Resolution

Connection Behavior (Multiplexing Disabled)

• Each client connection creates a new backend connection (1:1 mapping)
• Backend connections close only when frontend connections close
• High client concurrency leads to excessive backend connections
  
  Refer this techdoc for more information: Connection Multiplexing
  

 

Impact of Idle Connections

• Large number of idle frontend connections observed
• TCP keepalives keep these sessions active
• Backend connections remain open due to 1:1 mapping

 

Verification

Check open connections on Avi:

Virtual Service

 

show virtualservice <vs-name> detail | egrep "state|open_conns|open_requests|connections_dropped"

 

Pool Members

 

show pool <pool-name> server detail | grep -E "(\+--|ip_addr|se_uuid|open_conns|state)"

 

 

Resolution / Recommendation:

Enable connection multiplexing in the Application Profile associated with the Virtual Service.

Navigate to the Virtual Service → Edit → Application Profile, and enable the Connection Multiplexing option.

 

 

Enabling connection multiplexing:

• Decouples client and server-side connections
• Reuses backend connections
• Significantly reduces backend connection count
• Prevents source port exhaustion

This is strongly recommended for environments with:

• High connection rates
• Long-lived or keepalive-heavy traffic