NSX Edge detects pool members are DOWN due to Layer 4 connection problem

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Edge detects pool members are DOWN due to Layer 4 connection problem.
   
• In the NSX Edge log file, you see entries similar to:
  
  2017-02-23T10:19:05+00:00 NSX-Edge-0 loadbalancer[15780]: [htvs]: [local0.alert] Server LOADBALANCER/POOLMEMBER-01 is DOWN, reason: Layer4 connection problem, info: "General socket error (No buffer space available) at initial connection step of tcp-check", check duration: 0ms. 7 active and 0 backup servers left. 617 sessions active, 0 requeued, 0 remaining in queue.
  2017-02-23T10:19:06+00:00 NSX-Edge-0 loadbalancer[15780]: [htvs]: [local0.alert] Server LOADBALANCER/POOLMEMBER-02 is DOWN, reason: Layer4 connection problem, info: "General socket error (No buffer space available) at initial connection step of tcp-check", check duration: 0ms. 6 active and 0 backup servers left. 603 sessions active, 0 requeued, 0 remaining in queue.
  2017-02-23T10:19:25+00:00 NSX-Edge-0 kernel[]: [htvs]: [kern.info] TCP: out of memory -- consider tuning tcp_mem
  2017-02-23T10:19:27+00:00 NSX-Edge-0 loadbalancer[15780]: [htvs]: [local0.emerg] backend LOADBALANCER has no server available!
  2017-02-23T10:19:40+00:00 NSX-Edge-0 kernel[]: [htvs]: [kern.info] TCP: out of memory -- consider tuning tcp_mem
  2017-02-23T10:19:41+00:00 NSX-Edge-0 loadbalancer[15780]: [htvs]: [local0.notice] Server LOADBALANCER/POOLMEMBER-01 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
  2017-02-23T10:19:55+00:00 NSX-Edge-0 loadbalancer[15780]: [htvs]: [local0.notice] Server LOADBALANCER/POOLMEMBER-02 is UP, reason: Layer7 check passed, code: 0, info: "(tcp-check)", check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

NSX for vSphere

Cause

This issue occurs due to out of memory "net.ipv4.tcp_mem" kernel parameter.

Resolution

To work around this issue:

• In NSX for vSphere 6.2.7 and later versions:

Increase the "net.ipv4.tcp_mem" kernel parameter using PUT REST API for edge sysctl parameters.

Body:
<systemControl>
<property>sysctl.net.ipv4.tcp_mem=65536 63355 62333</property>
</systemControl>

• In earlier versions of NSX for vSphere 6.2.7:
  
  Increase the size of the edge as the parameter is not tunable.