GemFire: Replicated region not created on the server rejoining the cluster

Products

VMware Tanzu Gemfire

Issue/Introduction

A replicated region in the cluster was not created on a server that was restarted and rejoined the cluster. This region is not defined in server-cache.xml and is created and destroyed manually using gfsh while the cluster is running.

Environment

All supported versions of VMware Tanzu GemFire

Cause

A replicated region is only created on members that are part of the cluster at the time the region is created, unless the configuration is centrally persisted and replayed (for example, via the Cluster Configuration Service or GemFire dynamic regions).

Since this region was created manually with gfsh and is not in server-cache.xml or persisted via Cluster Configuration, a server that joins or restarts later has no knowledge of the region and therefore does not host it.

Resolution

The general recommendation is not to mix cache XML configuration and the Cluster Configuration Service for the same cluster, to avoid configuration drift and ambiguity.

To support dynamic creation of regions that are automatically applied to new and restarted members, regions should be created in gfsh while the Cluster Configuration Service (CCS) is enabled and running, so that the region definitions are stored and replayed to future members.

There is no supported way to “attach” an existing, running replicated region to a server that did not previously host it without creating the region on that server. To ensure that dynamically created regions are consistently hosted and persisted across the cluster, use gfsh create region with the Cluster Configuration Service enabled, rather than mixing cache XML and Cluster Configuration Service for the same configuration.

Option A – Non‑persistent replicated region

For a non‑persistent replicated region (no disk store):

You cannot add the existing in‑memory region to the restarted server without creating a region instance on that server.
The typical approach is:

Export the data from the existing region (for example, using gfsh export data or an application bulk read).
Destroy and recreate the region with the same name and type, ensuring all desired servers are online and included in the create region command, or that CCS is enabled so that new servers automatically create the region.
Import or reload the data into the recreated region.

There is no supported mechanism to “promote” an already-running non‑persistent replicated region so that it starts hosting on an additional server without a recreate workflow.

Option B – Persistent replicated region

For a persistent replicated region:

Each hosting member must have a compatible disk store and region definition; a server that never hosted the region does not have the required disk store contents and cannot simply begin hosting that persistent region.
The typical recovery or expansion pattern is:

Define the region and associated disk store on the target server so that its configuration matches the existing members.
Allow the new member to synchronize from existing peers where supported, or use an export/destroy/recreate/import procedure if you are changing which and how many members host the region.

As with non‑persistent regions, there is no direct way to add an already-running persistent replicated region to a previously non‑hosting server without defining the region and performing a synchronization or recreate/import workflow.

Additional Information

References