gfsh> shutdown --include-locators=true fails to wait for server departure and causes premature locator shutdown

Products

• VMware Tanzu Data Suite
• VMware Tanzu Data Intelligence
• VMware Tanzu Data Suite
• VMware Tanzu Gemfire

Issue/Introduction

When executing the gfsh > shutdown --include-locators=true command , the internal shutdown function dispatches the shutdown function to data nodes as an asynchronous operation and returns within milliseconds, immediately proceeding to shut down the locators while cache servers are still mid-close.

As a result, the cluster immediately proceeds to shut down the locators while the cache servers are still mid-close and attempting to drain their queues.

Impact: 

• Indefinite Hangs: Removing locators prematurely leaves cache servers unable to record clean departures from the membership network, which can trigger "stuck thread" or deadlocks and leave cache servers running as zombie processes.

• Network Churn: Cache servers trying to contact already-departed locators generate extensive Connection refused logs and continuous reconnect-retry loops.

• Data Instability: Forcing manual termination on hung nodes bypasses proper cluster exit checkpoints, increasing the risk of data inconsistency or DiskStoreException issues upon subsequent restarts.

Environment

 

Versions Affected:

• All 9.15.x
• All 10.0.x  until 10.0.8
• All 10.1.x until 10.1.8
• All 10.2.x until 10.2.5

 

Cause

A known product defect exists in the cluster shutdown sequencing. When a cluster-wide shutdown is initiated, via gfsh > shutdown --include-locators=true, a race condition can cause the Locators to terminate instantly while cache servers are still flushing data queues or gateway senders to disk.

The Underlying Mechanism

During member departure, GemFire iterates through a list of waiting components to deliver departure notifications.

• The Flaw: If any single component in that iteration sequence reports that it is already mid-close or shutting down, GemFire's walk terminates prematurely.

• The Impact: Any components positioned after that point in the walk fail to receive the necessary release signal. Because GemFire does not enforce a time limit on this specific wait condition, affected cache servers will experience an indefinite hang ("stuck threads") and become unresponsive zombie processes until manually terminated.

Risk Amplification

This defect requires at least two cache servers to be closing concurrently. Large-scale environments with numerous persistent partitioned regions or multiple gateway-sender queues expand the notification walk list significantly, drastically increasing the statistical probability of triggering this early-termination flaw.

 

 

 

Resolution

To prevent cluster hangs and stuck threads during shutdown on affected versions, do not use a single cluster-wide shutdown command. Instead, enforce a strict, sequential shutdown procedure:

1. Shut Down Cache Servers First:
   Execute the following command to target only the data nodes:

   gfsh> shutdown --time-out=600 --include-locators=false

2. Verify Process Termination: Actively monitor host operating systems (using jps or ps) to verify that all cache server JVM processes have completely terminated.

3. Shut Down Locators Last: Once you are 100% certain no cache servers remain running, issue the final command to stop the locators: 

   gfsh> shutdown --time-out=600 --include-locators=true

Note for Highly Vulnerable Environments: If your cluster continues to experience deadlocks due to multiple cache servers closing simultaneously, script the shutdown to loop over cache servers individually (stop one member, wait/poll until it fully departs, then move to the next) before proceeding to stop the locators.

Additional Information

This issue is tracked via GEM-19504. Fixes are scheduled to be available in the following upcoming releases:

• 10.2.6
• 10.3.1