GemFire: Firewall or similar network devices closes idle connections to GemFire

Products

• VMware Tanzu Gemfire
• Pivotal GemFire

Issue/Introduction

Firewall or similar network devices closes idle connections to GemFire, depending on the configuration of the network devices. 

Environment

Clients connects GemFire runs on Linux OS through a network device such firewall.

Cause

It is due to the network device such as firewall usually handle and terminates the idle connections that go through it depending on the configurations of the network devices. 

Resolution

Using KeepAlive in GemFire environment will fix this issue. 

TCP KeepAlive is a mechanism that helps prevent idle TCP connections from being prematurely closed by network devices such as firewalls. In VMware Tanzu GemFire, enabling TCP KeepAlive ensures that long-lived, idle connections remain active, which is crucial for stable cluster and client/server communications. GemFire uses the `gemfire.setTcpKeepAlive` system property to control TCP KeepAlive behavior.

When `gemfire.setTcpKeepAlive` is set to `true`, GemFire enables the `SO_KEEPALIVE` socket option for its connections. This causes the operating system to periodically send keepalive probes on idle connections.

By default, this property is set to `true`, so TCP KeepAlive is typically enabled out-of-the-box in most deployments.

 

As mentioned it uses OS feature which would require additional settings on OS level. Usually involves below settings, but it is recommended to consult internal OS and networking team to decide which settings and number to be set: 

net.ipv4.tcp_keepalive_time = 600      # Seconds before sending first keepalive probe (default: 7200)
net.ipv4.tcp_keepalive_intvl = 60      # Interval between probes (default: 75)
net.ipv4.tcp_keepalive_probes = 5      # Number of failed probes before closing (default: 9)

Additional Information

https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-gemfire/10-1/gf/managing-monitor_tune-socket_tcp_keepalive.html