Introduction to Tombstone GC Mechanism for Pivotal GemFire

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article applies to Pivotal GemFire 7.0.x to 8.0.x.

This document provides an introduction to the tombstone GC mechanism and control parameters introduced in GemFire 7.0 to 9.x.

Resolution

When consistency checking is enabled for a region, a GemFire member does not immediately remove an entry from the region when an application destroys the entry. Instead, the member retains the entry with its current version stamp for a period of time in order to detect possible conflicts with operations that have occurred. The retained entry is referred to as a tombstone. GemFire retains tombstones for partitioned regions and non-replicated regions, as well as, for replicated regions, in order to provide consistency.

A tombstone in a client cache or a non-replicated region expires after 8 minutes, at which point the tombstone is immediately removed from the cache. The system property, gemfire.non-replicated-tombstone-timeout, with a default of 480000 milliseconds, controls this timeout parameter.

A tombstone for a replicated or partitioned region expires after 10 minutes. The system property, gemfire.tombstone-timeout, with a default of 600000 milliseconds, controls this timeout parameter. Expired tombstones are eligible for garbage collection by the GemFire member. Garbage collection is automatically triggered after 100,000 tombstones of any type have timed out in the local GemFire member. Optionally, you can set the gemfire.tombstone-gc-threshold property to a value smaller than 100000 to perform garbage collection more frequently.

Besides these, there are a few other useful tombstone GC tuning parameters:

1. The interval to scan for expired tombstones in the queues:

   gemfire.tombstone-scan-interval = 60000 (default, milliseconds) 

2. The threshold percentage of free max memory that will trigger tombstone GCs. The default percentage is somewhat less than the LRU Heap evictor so that we evict tombstones before we start evicting cache data.

   gemfire.tombstone-gc-memory-threshold = 30 (default) 

3. Verbose tombstone logging for diagnosing tombstone problems.

   gemfire.TombstoneService.VERBOSE = false (default) 
   

Additional Information

1. An example to enable the tombstone gc parameter from gemfire server start script.

   -J-Dgemfire.tombstone-gc-threshold=20000 

2. You can also confirm tombstone GC stats from CachePerfStats:
   1. nonReplicatedTombstonesSize
   2. replicatedTombstonesSize
   3. tombstoneCount
   4. tombstoneGCCount