Offline compaction is not performing the same after upgrading from JDK8 to JDK11

Products

• VMware Tanzu Gemfire

Issue/Introduction

Offline compaction for persistent files in GemFire is run in a JVM separate to the GemFire members while the GemFire cluster is shut down. The details on the offline compaction command can be found here:
 

• https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-gemfire/10-1/gf/managing-disk_storage-compacting_disk_stores.html

If you are running offline compaction on big persistent files, you will usually specify a bigger heap to be created for the process:

gfsh>compact offline-disk-store —name=mydisk --disk-dirs=/disks/mydisk
--max-oplog-size=1024 -J=-Xmx8g

So why is the performance changing simply by upgrading from JDK8 to JDK11 without making any changes to GemFire?

In the above command, no garbage collector (GC) is specified. With JDK8, that means that the Parallel Collector will be used because that is the default collector and it works well for the compaction job. In JDK11, the default GC has changed to G1GC and, depending on the size of the heap configured for the compaction and the size of the files to compact, this can result in a lot of garbage collection and poor performance.

Resolution

To avoid this issue and to be able to run the offline compaction like it was run on JDK8, you can simply specify the Parallel Collector when running the following command:

gfsh>compact offline-disk-store —name=mydisk --disk-dirs=/disks/mydisk
--max-oplog-size=1024 -J=-Xmx8g —J=+UseParallelGC

If you want to get more information while running the offline compaction, you can also add GC logging parameters that are similar to the GC parameters that you usually have defined for the GemFire members in the cluster. For example:

gfsh>compact offline-disk-store —name=mydisk --disk-dirs=/disks/mydisk
--max-oplog-size=1024 --J=-Xms8g,-Xmx8g,-XX:+UseParallelGC,-Xlog:gc*:file=/logs/offlinecompaction/mydiskCompaction-verboseGC.log:time:filecount=10:filesize=100M

You can refer the following article for GC parameters with JDK11. There are some changes in GC parameters from JDK8 to JDK11 to note:
 

• https://knowledge.broadcom.com/external/article/294480/vmware-gemfire-garbage-collection-gc-log.html