Connectivity between clients and the GemFire cache server is lost

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:
Connectivity between clients and the GemFire cache server is lost.
 

• Clients behind firewalls lose connectivity with the GemFire cache server
• You experience IO exceptions on the cache server
• In the server-side log, you see messages similar to:

  CacheClientProxy[identity(client(5474:loner):1:d89cb8e0:ARLITE_BL_FWD_LDN_PRIM,
  connection=3,durableAttributes=DurableClientAttributes[id=; timeout=300]); port=46990; primary=true]: An unexpected IOException occurred so the proxy will be closed.java.net.SocketException: Connection timed out
  at java.net.SocketOutputStream.socketWrite0(Native Method)
  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
  at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

Environment

Cause

​In the GemFire client-server communication model, the client initializes the connection with the server by connecting a local, ephemeral port to a bound port on the server. Typically, a client-side firewall keeps track of these connections and allows traffic to flow from client-to-server and from server-to-client. However, with connections that have been inactive for along time, the firewall may forget or lose track of such connections and do not allow traffic to flow in from outside, for example, from the server to the client. Essentially, when the subscription connection remains idle for a relatively long period of time, the firewall may decide that it is defunct and closes the connection.

Resolution

The root cause of this issue is that the client-side firewall expires the client-server connection and treats inbound messages from the server as externally initiated traffic to a protected port and blocks them. To resolve tis issue, you can change the configuration on the firewall to allow such traffic. However, this may be difficult because the client-side port is assigned dynamically. Alternatively, you can increase the time that the firewall tracks outbound connections or ensure that there is regular traffic that prevents the firewall from closing the connection.

To workaround this issue by ensuring a regular traffic, use a small region for the express purpose of generating traffic. Specifically, create a small region on the server (possibly containing only a single key) and have the clients subscribe to that region/key. Then, use a thread on the server to routinely update that key. This generates regular traffic and prevents the firewall from deciding that the connection is dead. Also, this region can be used to test latencies and to keep clients alive. This approach works for Java and C# clients.