GemFire Latency Spikes and Socket Timeouts During Multi-Site Traffic Failback or Rerouting
search cancel

GemFire Latency Spikes and Socket Timeouts During Multi-Site Traffic Failback or Rerouting

book

Article ID: 449087

calendar_today

Updated On:

Products

VMware Tanzu Gemfire

Issue/Introduction

Following a scheduled maintenance window, site failback, or multi-site traffic rerouting event where client applications reconnect or ramp up traffic simultaneously:

1. Client Applications throw connection warnings and stack traces similar to:

WARN [org.apache.geode.cache.client.internal.ConnectionFactoryImpl] 
Function Execution Thread-XXXX - correlationId=YYYY 
Could not connect to: gemfire-server01.example.com:15001 
java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_382]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_382]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_382]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_382]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_382]
    at java.net.Socket.connect(Socket.java:613) ~[?:1.8.0_382]
    at org.apache.geode.distributed.internal.tcpserver.AdvancedSocketCreatorImpl.connect(AdvancedSocketCreatorImpl.java:106) ~[gemfire-tcp-server-10.1.0.jar:?]
    at org.apache.geode.internal.net.SCAdvancedSocketCreator.connect(SCAdvancedSocketCreator.java:55) ~[gemfire-core-10.1.0.jar:?]
    at org.apache.geode.distributed.internal.tcpserver.TcpSocketCreatorImpl.connect(TcpSocketCreatorImpl.java:63) ~[gemfire-tcp-server-10.1.0.jar:?]
    at org.apache.geode.distributed.internal.tcpserver.ClientSocketCreatorImpl.connect(ClientSocketCreatorImpl.java:58) ~[gemfire-tcp-server-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:100) ~[gemfire-core-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.ConnectionConnector.connectClientToServer(ConnectionConnector.java:79) ~[gemfire-core-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:128) [gemfire-core-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.pooling.ServerConnectionManagerImpl.createPooledConnection(ServerConnectionManagerImpl.java:198) [gemfire-core-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.pooling.ServerConnectionManagerImpl.createConnection(ServerConnectionManagerImpl.java:242) [gemfire-core-10.1.0.jar:?]
    at org.apache.geode.cache.client.internal.pooling.ServerConnectionManagerImpl.borrowConnection(ServerConnectionManagerImpl.java:277) [gemfire-core-10.1.0.jar:?]

2. Cluster Performance: Specific GemFire server nodes experience sudden, temporary latency spikes, high rates of client disconnects, and thread pool exhaustion during the initial reconnect wave.

Environment

All Supported versions of GemFire 

 

Cause

This issue is caused by the interaction of two main mechanisms during a mass client reconnection event:

1. TCP Listen Backlog Saturation (Thundering Herd)

When client applications are started or cut back to a primary cluster simultaneously, thousands of client application threads attempt to establish TCP connections at the exact same instant.

If client connection pools rely on default lazy connection creation (min-connections=0), this sudden influx of socket creation requests overwhelms the operating system kernel's TCP listen backlog queue. When the queue fills, the host OS drops incoming TCP SYN packets, causing client threads to fail with java.net.SocketTimeoutException: connect timed out before completing the initial socket handshake.

2. Un-Populated Single-Hop Metadata Maps (Multi-Hop Routing Overhead)

When cold client applications initialize, their local “single-hop bucket location metadata maps” (which track which server node hosts primary partitioned region buckets) are completely empty.

If full production traffic hits cold clients instantly, client requests cannot utilize single-hop routing and temporarily fall back to multi-hop operations. In multi-hop mode, receiving servers must proxy client requests across internal P2P network threads to the actual primary bucket owners. This temporary multi-hop proxying doubles CPU, thread, and network overhead across the cluster until client single-hop metadata maps are fully fetched and populated.

Resolution

1. Setting Client Minimum Connection Pool Size (min-connections)

Set min-connections on the client application pool to pre-allocate socket connections upon application startup, rather than establishing them lazily during traffic bursts.

  • Spring Data GemFire Example:
<gfe:pool id="clientPool" subscription-enabled="true" min-connections="10" max-connections="100"/>
  • Java API Example:
ClientCache cache = new ClientCacheFactory()
    .setPoolMinConnections(10)
    .setPoolMaxConnections(100)
    .create();

2. Gradually Ramping Up Client Traffic Instead of Bursting from 0 to 100

Avoid routing 100% of synchronous client traffic to a cluster in an instantaneous burst during site failback or cutback:

  • Gradually step up traffic volume over a 2- to 5-minute warm-up window instead of shifting from 0% to 100% all at once.
  • This allows client connection pools to establish stable sockets and populate their single-hop routing metadata maps under lower load, ensuring requests route directly to primary bucket servers without incurring internal multi-hop proxying overhead.

Additional Information

References

 

Powered by Wolken Software