Troubleshooting Tanzu GemFire on TAS: gfsh Keystore Prompts and WAN Replication Failures

Products

VMware Tanzu Gemfire

Issue/Introduction

When initializing or configuring cross-cluster WAN replication between two Tanzu GemFire service instances on Tanzu Application Service (TAS), administrators and developers may encounter the following connection and replication blockers:

Interactive gfsh Keystore Prompts: When connecting over GoRouter HTTPS (connect --url=https://...), gfsh interactively requests a Java KeyStore/truststore and password, even when --skip-ssl-validation is supplied and "tls-enabled": "false".
WAN Gateway Replication Failure: create gateway-sender commands fail to route traffic or establish connections to remote receivers because newly provisioned service instances share the default distributed_system_id: 0.

Environment

Platform: Tanzu Application Service (TAS) / Tanzu Platform for Cloud Foundry
Software: VMware Tanzu GemFire on TAS (v10.x runtime)
Client Tooling: Cloud Foundry CLI (cf), gfsh (GemFire Shell)

Cause

gfsh Interactive TLS Prompting
gfsh utilizes Java’s underlying HTTP client engine when executing connect --url=https://.... Because the GoRouter presents an HTTPS scheme, gfsh interactively prompts for key-store paths prior to completing the session setup.
Mechanism: Pressing Return/Enter feeds blank inputs into gfsh. If --skip-ssl-validation is supplied and "tls-enabled" is "false", gfsh falls back to the default JVM SSL context. Explicitly adding --use-ssl=false bypasses interactive prompting entirely.
Duplicate distributed_system_id Across Service Instances
By default, new Tanzu GemFire service instances provisioned on TAS assign distributed_system_id: 0. A GemFire Gateway Sender cannot establish a WAN connection or route data to a remote cluster that shares its own distributed_system_id.

Resolution

Step 1: Fix Service Instance IDs & Refresh Service Keys

Update the service instances so each cluster has a unique integer ID, then delete and recreate service keys to refresh the cached JSON metadata:

# 1. Reconfigure cluster IDs
cf update-service GemFire1 -c '{"distributed_system_id": 1}'
cf update-service GemFire2 -c '{"distributed_system_id": 2}'

# 2. Re-create service keys to refresh metadata
cf delete-service-key GemFire1 my-service-key1 -f
cf delete-service-key GemFire2 my-service-key2 -f

cf create-service-key GemFire1 my-service-key1
cf create-service-key GemFire2 my-service-key2

Step 2: Connect Non-Interactively via gfsh

Connect to each cluster's GoRouter URL without getting blocked by keystore prompts by explicitly setting --use-ssl=false:

gfsh> connect --url=https://<router-url>/gemfire/v1 \
      --user=<cluster_operator_user> \
      --password=<password> \
      --skip-ssl-validation \
      --use-ssl=false

Step 3: Establish WAN Gateway Senders

In gfsh, deploy the gateway senders targeting the remote cluster's new distributed_system_id:

A) On GemFire1 (Cluster ID 1): Point --remote-distributed-system-id to 2:

gfsh> create gateway-sender --id=sender-to-cluster2 --remote-distributed-system-id=2 --parallel=false --enable-persistence=true

B) On GemFire2 (Cluster ID 2): Point --remote-distributed-system-id to 1:

gfsh> create gateway-sender --id=sender-to-cluster1 --remote-distributed-system-id=1 --parallel=false --enable-persistence=true

Step 4: Attach Senders to Data Regions

Bind the gateway sender IDs to your target data regions:

gfsh> alter region --name=/myRegion --gateway-sender-id=sender-to-cluster2

Additional Information

References