GemFire Server Fails to Start with "Address already in use" During Stemcell Update

Products

• VMware Tanzu Platform Core

Issue/Introduction

During an Apply Changes operation in Ops Manager (specifically during a stemcell CVE patch or upgrade), the GemFire/PCC tile fails to deploy. Reviewing the logs for the affected GemFire server instance reveals the following:

• The process enters a crash loop.
  
  
• Error Message: org.apache.geode.GemFireIOException: Server port 40404 is already in use
  
  
• Exception: java.net.BindException: Address already in use
  
  
• The BOSH task fails during the "Updating Instance" phase for a specific GemFire member.

Environment

GemFire on Tanzu Platform

Cause

This issue occurs when a GemFire JVM process fails to shut down gracefully during the BOSH drain phase of a stemcell update.

If the process hangs or is unresponsive, BOSH may proceed with the VM update or process restart while the operating system still holds port 40404 in a LISTEN or TIME_WAIT state. When the new GemFire process attempts to start and bind to the default server port (40404), it fails because the port is still occupied by the previous stale process or the OS kernel has not yet released the socket.

Resolution

To resolve this issue and allow the "Apply Changes" task to proceed, follow these steps:

1. Identify the Occupied Port

SSH into the affected GemFire VM (e.g., bosh -d <deployment> ssh <job>/<index>) and check the status of port 40404:

bash

 

netstat -tpln | grep 40404

If a process ID (PID) is returned, a stale Java process is still running.

2. Terminate the Stale Process

If a PID was identified in the previous step, manually terminate the process to free the port:

bash

 

kill -9 <PID>

Wait a few seconds and run the netstat command again to ensure the port is no longer in a LISTEN state.

3. Recreate the Instance

In cases where the port is stuck in a kernel state (like TIME_WAIT) or the VM state is inconsistent, recreate the specific instance using the BOSH CLI:

bash

 

bosh -d <deployment_name> recreate <job_name>/<index>

Example: bosh -d service-instance_uuid recreate server/0

4. Resume Apply Changes

Once the instance is successfully recreated and the GemFire process is "running," return to Ops Manager and click Apply Changes to complete the tile update.