NSX Native Load Balancer Pool Members Intermittently "Down"

Products

• VMware NSX

Issue/Introduction

The NSX Native Load Balancer Virtual Servers are reporting a 'Degraded' status, with associated Server Pool members intermittently registering as 'Down.'"

Symptoms include:

• Intermittent connectivity issues to applications served by the Load Balancer (e.g., vRealize Automation, VMware Identity Manager).

• NSX Manager UI shows fluctuations in Pool Member health.

• Load Balancer debug logs (access logs) report:

  HLCK: Monitor server: check result: down, code:8
 Error.Reason: 'Datapath connection failure: Connection failed as no response from backend server'.

• Backend servers appear healthy and reachable via ping, but TCP health checks (e.g., Port 443) intermittently time out.

Environment

VMware vSphere ESXi

VMware NSX 4.2

Cause

The issue can be caused by physical NIC buffer exhaustion on the ESXi host where the Active NSX Edge node is residing.

Specifically, if one physical NIC (vmnic) in the host's uplink team experiences "Out of buffer" events, it can lead to dropped health check packets between the Load Balancer service and the backend pool members. Even if the backend server is operational, the dropped probes cause the Load Balancer to mark the members as "Down."

To verify this cause:

1. Identify the ESXi host where the Active Edge node is located.
2. Run the following command as root on the ESXi host to check for buffer issues: ./usr/lib/vmware/vm-support/bin/nicinfo.sh | less
3. Review the output for "Out of buffer" or "Rx-Drops" on the physical vmnics.

Resolution

To resolve this issue and restore stability to the Load Balancer health monitoring:

1. Isolate the Faulty NIC:

   • Temporarily administratively down the physical NIC reporting buffer errors to force traffic to the remaining healthy NIC(s) in the team: esxcli network nic down -n vmnicX
   • Observe if the Load Balancer pool member status stabilizes to "Up."
     
   • Re-enable the NIC: esxcli network nic up -n vmnicX

2. Hardware & Driver Maintenance:

   • Ensure the ESXi host physical NIC drivers and firmware are at the latest recommended versions per the VMware Compatibility Guide.
   • Investigate the physical switchport configuration (e.g., Flow Control, MTU) for any mismatches that could contribute to buffer pressure.

3. Backend Verification:

   • Ensure that backend application response times are within the "Timeout" and "Interval" parameters configured in the NSX Load Balancer Monitor.

Additional Information

Troubleshooting NSX Native Load Balancer

Setting the admin state up or down for a vmnic interface on ESXi