NSX Load Balancer VIP for Aria Operations intermittent TCP connection failure

Products

• VMware NSX

Issue/Introduction

• Intermittent failures when accessing the VMware Aria Operations VIP via the NSX Load Balancer  
• ICMP pings to the VIP are successful, TCP-based application traffic (HTTPS) frequently times out.
• Packet captures show SYN packets sent to the VIP but no return SYN-ACK (one-way communication)
• Load Balancer access logs contain errors indicating that server selection is failing due to "Max concurrent connections" being reached

root@host:/var/log/lb/e####ee5-####-4b93-####-87711971####/logs# tail -f access-####1694-####-49ef-####-d7cfdf0e####.log | grep -i 172.x.209.x
Operation.Category: 'LbAccessLog', Operation.Type: 'TCP', Lb.UUID: 'e####ee5-####-4b93-####-87711971####', Lb.Name: 'Loadbalancer-Test', Vs.UUID: '####1694-####-49ef-####-d7cfdf0e####', Vs.Name: 'vrops-https', Vs.Ip: '10.##.##.15', Vs.Port: '443', Pool.UUID: '5####aa0-####-####-####-4e0cd####c5f', Pool.Name: 'vrops-server-pool', PoolMember.IP: '-', PoolMember.Port: '-', Client.Ip: '172.x.209.x', Client.Port: '49889', Snat.Ip: '-', Snat.Port: '-', Error.Reason: 'Bad gateway. Server selection failed: max conn reached:Failed to get upstream config of sorry server'
Operation.Category: 'LbAccessLog', Operation.Type: 'TCP', Lb.UUID: 'e####ee5-####-4b93-####-87711971####', Lb.Name: 'Loadbalancer-Test', Vs.UUID: '####1694-####-49ef-####-d7cfdf0e####', Vs.Name: 'vrops-https', Vs.Ip: '10.##.##.15', Vs.Port: '443', Pool.UUID: '5####aa0-####-####-####-4e0cd####c5f', Pool.Name: 'vrops-server-pool', PoolMember.IP: '-', PoolMember.Port: '-', Client.Ip: '172.x.209.x', Client.Port: '49892', Snat.Ip: '-', Snat.Port: '-', Error.Reason: 'Bad gateway. Server selection failed: max conn reached:Failed to get upstream config of sorry server'

• The issue primarily impacts new connection attempts, while existing sessions may remain stable until they expire.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The Load Balancer Server Pool members have been manually configured with a strict Max Concurrent Connections limit (e.g., 10). Once this threshold is met, the Load Balancer stops forwarding new requests to that member, leading to dropped packets and connection timeouts.

Resolution

1. Log in to the NSX Manager UI.
2. Navigate to Networking > Load Balancing > Server Pools.
3. Locate and Edit the Server Pool used by the Aria Operations Virtual Server.
4. In the Pool Members section, identify the backend nodes:
   Node 1: 10.x.197.x
   Node 2: 10.x.196.x
5. Check the Max Concurrent Connections column.
6. Update the value from 10 to leave it blank to allow for unlimited concurrent connections.
7. Click Save.
8. Verify connectivity by establishing multiple concurrent sessions from a client machine.