Region import runs slowly because of large number of PdxTypes in GemFire

Products

• VMware Tanzu Gemfire

Issue/Introduction

When executing an import data command in GemFire, you may see log messages similar to the the following in log files and imports are running very slowly:

[info 2019/10/31 07:24:15.169 EDT server2 tid=0xb8e] Adding: PdxType[dsid=1, typenum=7198233 name=__GEMFIRE_JSON fields=[ nIoYasN:Object:0:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=-1]] [info 2019/10/31 07:24:15.172 EDT server2 tid=0xb8e] Adding: PdxType[dsid=1, typenum=6907799 name=__GEMFIRE_JSON fields=[ kth:Object:0:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=-1]] [info 2019/10/31 07:24:15.175 EDT server2 tid=0xb8e] Adding: PdxType[dsid=1, typenum=2322360 name=__GEMFIRE_JSON fields=[ tzhz:Object:0:idx0(relativeOffset)=0:idx1(vlfOffsetIndex)=-1]]

When unstructured JSON data causes a large number of PdxTypes to be created, this slows down performance and causes import operations to take a long time. 

Environment

Product Version: 9.7

Resolution

This article covers a couple options for improving performance in situations like this, with the goal being to reduce the number of PdxTypes:
 

• One option for reducing the number of PdxTypes is to set the system property gemfire.pdx.mapper.sort-json-field-names to "true".  You do this by passing it as a JVM parameter on server startup like this:

gfsh>start server --name=server1 --J=-Dgemfire.pdx.mapper.sort-json-field-names=true

• If you already have many unused PdxTypes, do a manual clean-up as explained in the following article: 
• https://knowledge.broadcom.com/external/article?articleNumber=294420

Additional information is available in the following resources:
 

• Blog: https://tanzu.vmware.com/developer/data/tanzu-gemfire/blog/removing-unused-pdxtypes/ 
• Code: https://github.com/boglesby/remove-unused-pdxtypes