How to export partition regions containing data in large size in Gemfire

Products

• VMware Tanzu Gemfire

Issue/Introduction

When upgrading from 8.2.x to 9.x, as part of the data backup procedure, there is a need to export region data to a file using the gfsh "gfs export data" command. In most of the use cases, there is no problem exporting reasonably sized regions, with an average size of less than 100GB, but the gfsh export command can fail for partitioned regions with larger sizes (e.g. 275gb) as servers may crash with a forced disconnect exception. This article shows a workaround to export data from Gemfire 8.2.x servers for partition regions containing greater than 100gb of data.

Environment

Cause

During the gfsh export data operations, the heap usage will keep increasing until it hits the critical threshold when servers will stop data operations, including any snapshot creating operation, even though the CPU usage might not very be high during this period.

To fix this issue, it is necessary to increase the heap size for cache servers (if this is an acceptable approach).

In Gemfire 8.2.x, there is no choice to export partition regions of large size from cache servers with gfsh using the --parallel option as there is in GemFire 9. However, the workaround presented in this article can be used.

In Gemfire 9.x, as the documentation mentions, for the case of partitioned regions, you can speed up the export process by using the --parallel option:

export data  --parallel --region=value --member=value --dir=value

Resolution

The attached ExportLocalDataFunction class and function execution service can be used to save the member's local data into members separately.

There are two approaches to run this functions:

a. Deploy ExportLocalDataFunction.java function in Gemfire cluster and then run it from a client application:

FunctionService.onRegion(this.region).withArgs("/directory/name").execute("ExportLocalDataFunction").getResult();

It'll create files in the input directory called: server_MEMBERNAME_region_REGIONNAME_snapshot.gfd. As long as the name is set in gemfire.properties, this will produce a different file for each member.

b. Similarly, the ExportLocalDataFunction.java function can also be deployed in Gemfire cluster and then can be run from one of the gfsh connections within the Gemfire cluster.

The following steps can be used as an example:

1. $ javac ExportLocalDataFunction.java 
2. $ jar cf exportfun.jar ExportLocalDataFunction.class
3. gfsh>deploy --jar=/home/gpadmin/apps/gemfire/classes/exportfun.jar
4. gfsh>list functions
5. gfsh>execute function --id=ExportLocalDataFunction --region=/TargetRegion --arguments=/directory/name
6. Then gfd files can be found in the directory that have been specified: server_ServerName_region_TargetRegion_snapshot.gfd

import com.gemstone.gemfire.cache.CacheFactory;
import com.gemstone.gemfire.cache.Declarable;
import com.gemstone.gemfire.cache.Region;

import com.gemstone.gemfire.cache.execute.Function;
import com.gemstone.gemfire.cache.execute.FunctionContext;
import com.gemstone.gemfire.cache.execute.RegionFunctionContext;

import com.gemstone.gemfire.cache.snapshot.RegionSnapshotService;
import com.gemstone.gemfire.cache.snapshot.SnapshotOptions.SnapshotFormat;

import com.gemstone.gemfire.cache.partition.PartitionRegionHelper;

import java.io.File;

import java.util.Properties;

public class ExportLocalDataFunction implements Function, Declarable {

  private final String memberName;
  
  public ExportLocalDataFunction() {
    this.memberName = CacheFactory.getAnyInstance().getName();
  }
  
  public void execute(FunctionContext context) {
    // Get directory name
    String  directoryName = (String) context.getArguments();

    // Get local data set
    RegionFunctionContext rfc = (RegionFunctionContext) context;
    Region lds = PartitionRegionHelper.getLocalDataForContext(rfc);
    
    // Create the file
    String fileName = "server_" + this.memberName + "_region_" + lds.getName() + "_snapshot.gfd";
    File file = new File(directoryName, fileName);

    // Export local data set
    RegionSnapshotService service = lds.getSnapshotService();
    try {
      System.out.println(Thread.currentThread().getName() + ": Exporting " + lds.size() + " entries in region " + lds.getName() + " to file " + file.getAbsolutePath() + " started");
      service.save(file, SnapshotFormat.GEMFIRE);
      System.out.println(Thread.currentThread().getName() + ": Exporting " + lds.size() + " entries in region " + lds.getName() + " to file " + file.getAbsolutePath() + " completed");
    } catch (Exception e) {
      context.getResultSender().sendException(e);
      return;
    }

    context.getResultSender().lastResult(true);
  }
  
  public String getId() {
    return getClass().getSimpleName();
  }

  public boolean optimizeForWrite() {
    return true;
  }

  public boolean hasResult() {
    return true;
  }

  public boolean isHA() {
    return false;
  }

  public void init(Properties properties) {
  }
}