Gemfire locators/members not able to communicate to each other

Products

• VMware Tanzu Gemfire

Issue/Introduction

When the first locator/gemfire process is started, it initializes successfully across all VMs/hosts. However, when attempting to start a second locator process, the new process fails to launch. The error message indicates that it is unable to communicate with the first locator, leading to the process shutting down automatically.

Environment

This issue occurs across multiple environments, including Linux and virtual machines (VMs). When the locator process shuts down, you will encounter the following message in the logs:

Locator is exiting due to an exception
org.apache.geode.SystemConnectException: Unable to join the distributed system in 25773ms
  at org.apache.geode.distributed.internal.DistributionImpl.start(DistributionImpl.java:184)
  at org.apache.geode.distributed.internal.DistributionImpl.createDistribution(DistributionImpl.java:220)
  at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:464)
  at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:497)
  at 

Cause

When addressing this issue, it is important to consider multiple potential causes. To begin troubleshooting, please ensure the following:

1. Verify that there are no network connectivity issues.

2. Check for any network or firewall-related problems within the environment.

Resolution

To ensure proper operation of GemFire processes, it's essential to configure firewall settings to allow both inbound and outbound traffic on the specific ports used by gemfire processes.

Instructions:

1. Allow Inbound and Outbound Traffic:
   Ensure that the firewall settings permit both inbound and outbound traffic for the ports utilized by the GemFire processes.

2. Configure P2P Communications:
   When setting up Peer-to-Peer (P2P) communications across a firewall, it is crucial to:

   • Specify the TCP port for each GemFire process.
   • Make sure that UDP traffic is permitted through the firewall as well.

 

Additional Information

Applicable for GemFire versions : All