GemFire: Set EXPIRY_THREADS when using expiration feature

Products

• VMware Tanzu Gemfire

Issue/Introduction

We have discovered an issue in current GemFire products where it is possible to lose expiration related messages.   The symptoms will be that expiration will potentially stop expiring entries, driving higher entry counts and higher heap consumption, resulting in potential GC issues, membership issues, and negative business impact.

Environment

Product Version: 9.15
OS: ALL

Resolution

We are already investigating potential candidate fixes for this issue.  That said, we already do have an easy way to greatly reduce chances of being impacted by any such lost expiration message.

GemFire has a system property:   gemfire.EXPIRY_THREADS

If you set this value to any value greater than 1, you would be protected from any single lost message.   Essentially, the higher you set the number of expiration threads, the more protection you have from being impacted.   In other words, if one such thread becomes blocked irreparably due to a lost message, you will still have other expiry threads continuing to expire entries.

GemFire also has plans to change the default of gemfire.EXPIRY_THREADS going forward, purely as an additional protection, ever as we will also fix our known lost message issue.

The recommendation is to set gemfire.EXPIRY_THREADS=3 going forward to protect you, until we alter the default, and eliminate our known timing/race condition issue driving this bad behavior.