GemFire- Leaking P2P Message Reader Threads

Products

• VMware Tanzu Gemfire

Issue/Introduction

Symptoms:
The number of P2P message reader threads can grow indefinitely over time for certain configurations. This can certainly be problematic for a variety of reasons related to resource consumption. This issue is not a concern for conserve-sockets=true environments.

Environment

Cause

For performance reasons, customers sometimes use various configuration settings to avoid hitting various limitations with respect to the number of threads or timing out of threads. While this can be beneficial, it can lead to a situation where we leak GemFire P2P message reader threads. The combination of settings to be avoided is discussed in this article.

The properties to be discussed include settings for each of the following:

• socket-lease-time=0
• max-threads=0
• conserve-socket=false
• gemfire.IDLE_THREAD_TIMEOUT

Root Cause Analysis

This is the result of a product issue that continues to exist with GemFire.   There is no imminent resolution pending given the various workarounds that resolve this issue. Basically, when trying to avoid the costs of starting threads and tearing down connections or throttling the system, customers may use settings where we never timeout a thread and never set a limit on the maximum number of threads. This can result in a leak of P2P message reader threads because the mechanisms in place to limit the number and clean them up have been overridden

Resolution

Any of the following options can be used to eliminate the leaking P2P threads: 

1. Do not use socket-lease-time = 0.  The default is 60 seconds. If you feel that keeping sockets around longer than this is warranted, set this property to some higher value that will clean up the idle socket in time.  The maximum value is 10 minutes (600000 milliseconds)  

2. Set max-threads to a non-zero value. Setting max-threads to 0 causes the server to use a new thread for every client connection. When used in conjunction with socket-lease-time=0, the result can be many threads and sockets that do not get cleaned up properly when no longer in use. Setting max-threads greater than 0 will limit the number of threads that can service client connections, thus reducing the leaking threads.

3. The system property gemfire.IDLE_THREAD_TIMEOUT controls how long an idle thread will wait, in milliseconds, before it is removed from its thread pool. The default time is 30 minutes.