"IllegalStateException: A Gateway Receiver already exists on this member" error after cache server tries to restart in Tanzu GemFire for VMs

Products

• VMware Tanzu Gemfire

Issue/Introduction

Tanzu GemFire for VMs automatically creates gateway receivers on each cache server node during startup so that the hostname for clients is properly configured. Hence, the cluster configuration service will have no only a cluster.xml, but also a cache.xml file for each cache server containing an entry similar to:

<gateway-receiver hostname-for-senders="xxxxx.server.pas-services.service-instance-yyyyy.bosh" manual-start="false"/> 

When a user manually creates gateway receivers using the gfsh create gateway-receiver command, without specifying a member or group, an entry similar to the one below will be added to the cluster configuration cluster.xml:

<gateway-receiver manual-start="false"/>

When a cache server tries to start, it ends up with configuration that is a combination of these two XML configurations, the cluster.xml and the node specific cache.xml, which will result in the cache server having two gateway receivers defined.

This causes the server to fail with the error:

java.lang.IllegalStateException: A Gateway Receiver already exists on this member

Environment

Product Version: 1.11

Resolution

Please note that this issue can happen if the VM(s) are continuously failing and the service metrics process is flapping. In that scenario, recreate the failing VM(s) and check if the issue is still happening during the restart. If the issue is not resolved, then proceed with the steps below.

To resolve this issue, follow these steps:

1. First, export the existing cluster config xml:

gfsh> export cluster-configuration --xml-file=ClusterConfig.xml

2. Edit the xml file and remove the line:

<gateway-receiver manual-start="false"/>

3. Then import the corrected xml file as "STAGE":

gfsh> import cluster-configuration --xml-file=configs/ClusterConfig.xml ‑‑action=STAGE

4. Finally, restart the cache servers. Due to the startup failures, this will likely be automatic with Tanzu GemFire for VMs.