GemFire: Entries Remaining on Disk After Server Recovery in GemFire Partitioned Regions with Overflow or Persistence

Products

• VMware Tanzu Gemfire
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite
• VMware Tanzu Data Intelligence

Issue/Introduction

After recovering or restarting servers that previously had their disk stores cleared, users may observe that entriesOnlyOnDisk remains non-zero even after the servers have rejoined the cluster and redundancy has been restored.

The expectation is often that these entries should move into memory automatically, but they do not.

Environment

Region Type: Partitioned regions with overflow and/or persistence

Cause

This behavior is expected in GemFire’s design for partitioned regions using overflow or persistence.

When a member that does not have persistent data rejoins the cluster:

1. The region and bucket metadata are restored to reestablish redundancy and bucket ownership.
2. However, data entries stored on disk are not automatically faulted into memory (heap).
3. Entries remain on disk until they are explicitly accessed, through:
• Client read or query operations,
• Region iteration, or

Therefore, the entriesOnlyOnDisk metric remains greater than 0 until these entries are accessed.

Resolution

This is normal and does not indicate an error. The following approaches can be considered:

Option 1 – Manual Access (Not Recommended for Large Datasets)

• Start the servers where data remains on disk.
• Access the entries region by region to load them into memory.

However, this method can strain the cluster and prolong recovery, especially when the dataset is large.

Option 2 – Allow Natural Access (Recommended)

• Allow your applications and client traffic to naturally access the data over time.
• Entries will be faulted into memory on demand, as they are read or updated.
• You may notice minor latency (in milliseconds) during the first access, but this is the safest and most efficient recovery method.