GemFire: Not able to start gfsh and locators due to classloader isolation after upgrading to 10.x

Products

VMware Tanzu Data Suite Pivotal GemFire VMware Tanzu Gemfire

Issue/Introduction

After upgrading to GemFire 10.x, users may be unable to start GFSH, and consequently, locator/server processes fail to start.

Environment

This issue typically occurs after an in-place upgrade from GemFire 9.x to 10.x, especially when the upgrade is performed without following the official documentation:

https://techdocs.broadcom.com/us/en/vmware-tanzu/data-solutions/tanzu-gemfire/10-0/gf/getting_started-upgrade-upgrade-v9-to-v10.html

Cause

The issue is caused by changes in classloader hierarchies and isolation mechanisms introduced in GemFire 10.x.
In GemFire 10, JAR files are deployed using a classloader isolation model. Each deployed JAR is loaded into its own isolated classloader. These classloaders do not share resources with each other or the system by default. A deployment will first attempt to load classes from its own classloader before looking elsewhere.
Classloader isolation is the default and recommended setting for all new GemFire 10 implementations. In contrast, chained classloading was the default in GemFire 9.x and is still available to support legacy applications.

Resolution

Troubleshooting:

To verify whether the issue is caused by classloader isolation, you can temporarily disable it using the --disable-classloader-isolation=true flag when starting members via GFSH.
This step can help identify if classloader isolation is the root cause. If disabling it resolves the issue, it confirms the isolation behavior is likely responsible.Example of the tag usage:

Start gfsh with the tag:

DISABLE_CLASSLOADER_ISOLATION=true /path/to/gfsh

Start locator/server with the tag:

gfsh>start locator ... --disable-classloader-isolation

gfsh>start server ... --disable-classloader-isolation

Resolution:

Once the issue is identified, resolve it by following these best practices:
1. Avoid setting the system-level CLASSPATH when using GemFire 10.x. While this approach may have worked in 9.x (or in 10.x with classloader isolation disabled), it leads to errors when isolation is enabled.   
2. Unset the CLASSPATH environment variable. If you need to include custom JARs, use the AUTOMATIC_MODULE_CLASSPATH environment variable instead.
3. Use the --classpath flag to refer to required JARs when starting members.

Validation:

After starting the locator or server, you should see that classloader modules are properly isolated but interdependent, as shown in the logs:

gfsh:

###@gf-00:~/gf/cluster-10.0.6$ unset CLASSPATH
###@gf-00:~/gf/cluster-10.0.6$ $GEMFIRE/bin/gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  / 
 / /__/ / ____/  _____/ / /    / /  
/______/_/      /______/_/    /_/    10.0.6

Monitor and Manage Tanzu GemFire
gfsh>start locator --name=locator1 --classpath=/path/to/gemfire-greenplum-4.0.1/gemfire-greenplum-4.0.1.jar
Starting a GemFire Locator in /path/cluster-10.0.6/locator1...
.
Locator in /path/cluster-10.0.6/locator1 on gf-00[10334] as locator1 is currently online.

Log:

[info 2025/05/20 20:42:40.767 PDT locator1 <main> tid=0x1] ClassLoader Modules:
- Module Name: _automatic_
-- Resources:
--- gemfire-greenplum-4.0.1.jar
--- /tmp/gemfire-extensions14555609290833264378/_automatic_/main/lib
-- Dependencies:
--- module java.base
--- local resources
--- module default_application
--- module gemfire
--- module java.se

- Module Name: default_application
-- Resources:
--- /path/to/vmware-gemfire-10.0.6/moduleDescriptors/default_application/main/lib
-- Dependencies:
--- module java.base
--- local resources
--- module _automatic_

- Module Name: gemfire
-- Resources:
--- gemfire-server-all-10.0.6.jar
......
-- Dependencies:
--- module java.base
--- local resources
--- module default_application
--- module java.se
--- module jdk.unsupported
--- local loader org.jboss.modules.JDKSpecific$1@309e345f