GemFire: Function Executions and conserve-sockets behavior
Article ID: 294468
Updated On:
Products
VMware Tanzu Gemfire
Issue/Introduction
Most customers set conserve-sockets=false in GemFire environments to avoid needing to share sockets because there are times this can lead to distributed deadlock incidents.
That said, many customers are unaware that even with conserve-sockets set to false, function executions do not use this setting and defaults to conserve-sockets=true behavior, regardless of the conserve-sockets setting.
Basically, when conserve-sockets=false behavior was being incorporated as an option into the GemFire product, it was not incorporated into the behavior of function executions.
Due to this behavior, at times depending on the customer function code and environment, we find evidence of function executions getting blocked due to having this conserve-sockets=true behavior.
This is an issue that requires incorporating some changes in function execution code to avoid this blocking activity by essentially altering the behavior for function executions using the following methods.
That said, many customers are unaware that even with conserve-sockets set to false, function executions do not use this setting and defaults to conserve-sockets=true behavior, regardless of the conserve-sockets setting.
Basically, when conserve-sockets=false behavior was being incorporated as an option into the GemFire product, it was not incorporated into the behavior of function executions.
Due to this behavior, at times depending on the customer function code and environment, we find evidence of function executions getting blocked due to having this conserve-sockets=true behavior.
This is an issue that requires incorporating some changes in function execution code to avoid this blocking activity by essentially altering the behavior for function executions using the following methods.
Environment
Product Version: 9.10
OS: Linux
OS: Linux
Resolution
You can override the conserve-sockets setting for individual threads. These methods are in org.apache.geode.distributed.DistributedSystem:
setThreadsSocketPolicy
Sets the calling thread’s individual socket policy, overriding the policy set for the application as a whole. If set to true, the calling thread shares socket connections with other threads. If false, the calling thread has its own sockets.
releaseThreadsSockets
Frees any sockets held by the calling thread. Threads hold their own sockets only when conserve-sockets is false. Threads holding their own sockets can call this method to avoid holding the sockets until the socket-lease-time has expired.
Example function execution usage:
If you are not doing so and you have many functions executing concurrently, you may incur some unforeseen cost due to GemFire where once you surpass the default number of concurrent function executions there is some additional cost to throttle those executions.
The default for MAX_FE_THREADS is the greater of the number of processors * 4, or 16. This is often completely insufficient for customers that are heavy users of GemFire functions.
setThreadsSocketPolicy
Sets the calling thread’s individual socket policy, overriding the policy set for the application as a whole. If set to true, the calling thread shares socket connections with other threads. If false, the calling thread has its own sockets.
releaseThreadsSockets
Frees any sockets held by the calling thread. Threads hold their own sockets only when conserve-sockets is false. Threads holding their own sockets can call this method to avoid holding the sockets until the socket-lease-time has expired.
Example function execution usage:
public void run() {
DistributedSystem.setThreadsSocketPolicy(false);
try {
// do your work
} finally {
DistributedSystem.releaseThreadsSockets();
}
}
Finally, if you are a heavy user of GemFire functions, you likely want to consider using the system property gemfire.MAX_FE_THREADS and set it to some high appropriate value. If you are not doing so and you have many functions executing concurrently, you may incur some unforeseen cost due to GemFire where once you surpass the default number of concurrent function executions there is some additional cost to throttle those executions.
The default for MAX_FE_THREADS is the greater of the number of processors * 4, or 16. This is often completely insufficient for customers that are heavy users of GemFire functions.
Feedback
Yes
No
Powered by
