Gemfire: Addressing client latency and AllConnectionsInUseException

Products

• VMware Tanzu Gemfire

Issue/Introduction

 When you get the  AllConnectionsInUseException along with client latency you need to review and tune client-side pool configuration. This article provides an overview of key factors to keep in mind while reasoning about client latency issues and tuning client-side pool configuration.

 

Environment

Gemfire v10+

Resolution

Client-side pool configuration and tuning is an iterative process.There is never an "optimal" or single correct value.  These are guidelines to follow and testing in an identical test cluster is advised.

min-connections-per-server - The more important setting is the min-connections-per-server. This should be set at the typical concurrent operations for a client instance.

For  a client with 10 concurrent threads, a min value of 10 would guarantee that no thread has to wait to establish a new connection. This however has impact on the servers since each client may have more open and idle connections. If the min value is too low and the workload is bursty you could end up needing connections that aren't already established, which increases the latency. If the max is higher it may prevent this a little but only if the bursts are within the idle connection timeout, otherwise you just end up with connection churn. In other words, a connection churn occurs when a burst of concurrency requires more connections but idle out shortly after and the cycle repeats. This churn can be expensive and increases latency, especially if using TLS.

max-connections-per-server - The main reason to use max-connnections or max-connections-per-server on the client is to limit the number of connections and concurrent requests the clients are making to the servers. Increasing the max-connections on the client could help  if servers have capacity for the increased load. If the servers already have high numbers of connections, threads, or high CPU increase server capacity to deal with the increased load.

In practice, start with min-connections-per-server=1 and max-connections-per-server=10 and then test. Turn on stats on the client and then monitor the number of connections that are really used. If you notice that the number of connections is close to max, add 10% to the max and test again. Also, don't set min-connections=max-connections,  as this would keep connections  open that are not required and that affects both the cluster and client resource usages. Also do not set max-connections to unlimited.

Additional Information

Please refer to this KB for additional information on client timeouts and pool configuration. Related documentation on pr_single_hop is here.