GemFire: Monitoring client connection pools and client health
Article ID: 428282
Updated On:
Products
Issue/Introduction
- How can I monitor the number of currently established client connections and any pending connections?
- Is there a way to access client-side statistics through the client API or have them printed in the log file?
Environment
All Supported GemFire versions
Cause
In VMware GemFire client-side statistics are not exposed directly through Pool or ClientCache configuration objects. These configuration objects describe how to connect (intent), whereas the Statistics subsystem and management interfaces (JMX, gfsh, .gfs archives) describe what is happening (metrics).
Resolution
If you have limited filesystem access and cannot easily retrieve .gfs statistics archives, there are two main approaches:
- Programmatic access to client-side statistics from within the client JVM.
- Server-side client health information via JMX (CacheServerMXBean) or gfsh.
- Programmatic client-side statistics (PoolStats, ClientStats)
Client connection pool statistics on the client are exposed via the Statistics API, not via Pool or ClientCache directly. Client pool statistics are grouped under the type name PoolStats, and client messaging metrics under ClientStats / ClientSendStats.
High-level steps:
- Obtain a StatisticsFactory from the GemFire system in the client.
- Query statistics of type PoolStats and ClientStats for your pool name.
- Read fields such as current connections, messages in progress, and operation counts (for example, gatewayBatchSendsInProgress, getAllSendsInProgress, etc.).
This requires custom code in the client and is the primary way to get “client-side” values for active connections and in‑flight operations without relying on .gfs files.
- Enabling statistic sampling and archives
Even if you cannot routinely pull the .gfs archive, it is still recommended to enable statistics sampling so the metrics exist and can be inspected when needed.
In gemfire.properties on the client:
- statistic-sampling-enabled=true
- statistic-archive-file=clientStats.gfs (path as appropriate)
- Optionally: statistic-sample-rate=2000 (milliseconds), archive-file-size-limit, archive-disk-space-limit.
This does not automatically print stats to the logs, but it ensures that both the programmatic API and tools (e.g., gemfire stats, gfsh show metrics) have data to work with.
- Logging metrics from the client (custom thread)
GemFire does not periodically dump detailed statistics to the client log at normal log levels. To see selected metrics in logs, you typically:
- Enable statistics sampling (as above).
- Implement a small scheduled task (for example, ScheduledExecutorService) in the client application that:
- Reads the desired PoolStats / ClientStats fields via the Statistics API.
- Logs them at a chosen interval (for example, every 60 seconds).
This approach is generally preferred over globally increasing log verbosity, because it gives you fine‑grained control over which statistics are written and at what rate.
Server-side client health via JMX and gfsh
Client “health” and connection counts can also be obtained on the server side, for connected clients.
- The server’s CacheServerMXBean exposes client health and statistics via methods such as showAllClientStats() and per‑client queries.
- These include information such as CPU process time, number of puts/gets, ping time, and connected/idle status for each client.
Usage considerations:
- Invoke showAllClientStats() on each cache server MBean to retrieve stats for the clients connected to that server.
- These statistics are server-side views of client health; they apply to connected clients and do not expose pending connection attempts or internal client‑pool wait queues on the client.
You can also obtain high-level connection counts via:
- gfsh show metrics --member=<server> to view fields such as currentClients and currentClientConnections in the cache-server section.
- External tooling or dashboards that query the JMX manager for these MBeans.
Example: Spring-based JMX client
A concrete example of using JMX from a Spring application to query GemFire health is available here:
This shows how to connect to GemFire JMX and invoke management operations from a Java/Spring application.
Additional Information
Important limitations and clarifications
- Client-side statistics such as PoolStats and ClientStats are accessible inside the client JVM through the Statistics API when statistics sampling is enabled; they are not directly exposed as simple getters on Pool or ClientCache.
- JMX CacheServerMXBean and gfsh show metrics provide server-side views of client health (connected clients, latency, operation counts), not the client’s internal pending‑connection queues.
- Monitoring “waiting connections” specifically on the client side requires custom code that inspects the relevant client statistics and/or threads; there is no single JMX or gfsh command that lists “pending client pool connections.”
References:
Feedback
