Gemfire: import data from GPDB with connector

Products

• VMware Tanzu Gemfire

Issue/Introduction

This guide will show you how to configure the Gemfire with GPDB connector so that GPDB data can be imported into a Gemfire region.

Environment

Product Version: 9.10

Resolution

Prerequisites:
A running GPDB instance and, for this example, is should have a database called test and table called Child.

In this example, we use Gemfire 9.10.5. 

1. Install the Gemfire per the following guide: 
https://docs.vmware.com/en/VMware-GemFire/9.10/gf/getting_started-installation-install_intro.html

2. Create a new directory named "extensions" under the Gemfire installation directory
mkdir -p $GEMFIRE_HOME/extensions/

3. Copy all the jar files from gemfire-greenplum-3.5.0.tar.gz into this extensions directory
cd $GEMFIRE_HOME/extensions/
tar xzvf gemfire-greenplum-3.5.0.tar.gz

gemfire-greenplum-3.5.0-javadoc.jar
gemfire-greenplum-3.5.0.jar
open_source_license_Connector_for_VMware_Tanzu_Greenplum_and_VMware_Tanzu_GemFire_3.5.0_GA.txt

4. Start up a locator and a server with the gfsh
5. Configure PDX serialization

gfsh>configure pdx --read-serialized=true --auto-serializable-classes=io.pivotal.gemfire.demo.entity.*
The command would only take effect on new data members joining the distributed system. It won't affect the existing data members
read-serialized = true
ignore-unread-fields = false
persistent = false
PDX Serializer = org.apache.geode.pdx.ReflectionBasedAutoSerializer
Non Portable Classes = [io.pivotal.gemfire.demo.entity.*]
Cluster configuration for group 'cluster' is updated.

gfsh>

6. Create jndi-binding

gfsh>create jndi-binding --name=datasource --type=SIMPLE --jdbc-driver-class="org.postgresql.Driver" --username="gpadmin" --password="changeme" --connection-url="jdbc:postgresql://X.X.X.X:5432/test"
Member  | Status | Message
------- | ------ | -----------------------------------------------
server1 | OK     | Created jndi-binding "datasource" on "server1".

Cluster configuration for group 'cluster' is updated.

gfsh>

Please replace the X.X.X.X with your GPDB server's IP. Note that test URI element refers to the GPDB database described in the prerequisites and you will need to use the associated credentials for that database.

7. Configure the gpfdist

gfsh>configure gpfdist-protocol --port=8000
Member  | Status | Message
------- | ------ | ------------------------------
server1 | OK     | gpfdist listening on port 8000

Cluster configuration for group 'cluster' is updated.

gfsh>

8. Create a region in Gemfire

gfsh>create region --name=Child --type=REPLICATE
Member  | Status | Message
------- | ------ | ------------------------------------
server1 | OK     | Region "/Child" created on "server1"

Cluster configuration for group 'cluster' is updated.

gfsh>

8. Specify the mapping 

gfsh>create gpdb-mapping --region=/Child --data-source=datasource --pdx-name="io.pivotal.gemfire.demo.entity.Child" --table=child --id=id,parent_id
Member  | Status | Message
------- | ------ | -------
server1 | OK     | SUCCESS

Cluster configuration for group 'cluster' is updated.

gfsh>

9. Import the data from GPDB

gfsh>import gpdb --region=/Child
GemFire entries imported : 101000
Duration                 : 8.81s

gfsh>

You can check the server log for this import operation:

[info 2023/08/16 08:09:25.510 UTC <Function Execution Processor2> tid=0x39] 1692166259-0000000048 40 execute: Started transaction.

[info 2023/08/16 08:09:29.617 UTC <Function Execution Processor2> tid=0x39] 1692166259-0000000048 4147 execute: Commited transaction.

[info 2023/08/16 08:09:29.618 UTC <Function Execution Processor2> tid=0x39] Operation 'eb25ac7f-ec8e-4f4b-b3c4-ee2ba327eedf' processed 101000 rows (1185576 bytes)

This shows that configuration of the GPDB connector for Gemfire is finished and import data from GPDB is successful.