How to solve data out-of-sync issues in WAN configured VMware Tanzu GemFire

Products

• VMware Tanzu Gemfire

Issue/Introduction

This article describes the necessary steps to re-sync data between two TanzunGemFire clusters that are connected via WAN and have become out-of-sync. In certain situations, due to connectivity issues between the clusters or an outage on one side, WAN-connected clusters can become out-of-sync. We will discuss how to resolve such situations using an export-import strategy.

Resolution

The steps listed below need to be followed this exact sequence.
 

Step 0

Using your understanding of data and use case, pick the cluster that holds the most valid data to be the source.

Step 1

Stop all Gateway Receivers and redirect traffic away from the target cluster.

•  https://docs.vmware.com/en/VMware-GemFire/10.1/gf/tools_modules-gfsh-chapter_overview.html 

Caution: Stopping the receivers will cause the senders on the other cluster to queue events, so not having overflow configured could lead to out-of-memory situations.
 

Step 2

Export region data from the source cluster (as determined in Step 0).

•  https://docs.vmware.com/en/VMware-GemFire/10.1/gf/tools_modules-gfsh-chapter_overview.html 

Note: You need to repeat this step for every region. For partitioned regions only, using --parallel exports local data on each node to a directory on that machine, which can be more efficient.
 

Step 3

Import region data to the target cluster, assuming the region is already present on that cluster. In a disaster recovery case, it’s assumed to be an empty cluster that needs to get the latest data from the source cluster.

•  https://docs.vmware.com/en/VMware-GemFire/10.1/gf/tools_modules-gfsh-chapter_overview.html 

  • In this step, you will need to take the exported files (*.gfd) to the respective hosts in the target cluster.

  • Run the import command.

Note: If you used a serial export, you only need to run this on the one node holding the exported .gfd file. In the case of parallel exports, you need to repeat this import step for every region on each cache server (refer to the documentation link above).

Step 4

Ensure the import finishes successfully

It's very important to make sure the import process finishes uninterrupted. To verify a successful import, use the gfsh describe region command to make sure it has the expected size.

Step 5

Start the Gateway Receivers on the target cluster where the data is now successfully imported and all members are online.
 

•  https://docs.vmware.com/en/VMware-GemFire/10.1/gf/tools_modules-gfsh-chapter_overview.html 

Step 6

Finally, enable the traffic to the target cluster.