Capturing Network Traffic Between GemFire Locator and Server Using TCPDump

Products

• VMware Tanzu Gemfire

Issue/Introduction

When troubleshooting network communication issues between a GemFire Locator and Server Members, it may be necessary to capture and analyze the TCP traffic between them. This can help identify problems like dropped connections, latency, port blocks, or protocol mismatches.
This article provides steps to capture traffic using tcpdump between a GemFire Locator and its Servers.

Environment

VMware Tanzu GemFire 9.15 and 10.x

Cause

Network or firewall issues, misconfigured ports, or JVM-level problems may impact communication between GemFire components. Packet capture helps in low-level debugging.

Resolution

1. Identify Ports Used by GemFire
By default, Locator communicates over port 10334 (can vary if customized)
Server members may communicate over various ports including JMX or other internal distributed system ports.
You can confirm ports via the gemfire.properties, gfsh output, or netstat.

2. Run tcpdump on the Locator and server in parallel

   a. Run tcpdump on the Locator
        Example: Capture all traffic between locator and a specific server (192.168.1.50) on port 10334:
        sudo tcpdump -i eth0 host 192.168.1.50 and port 10334 -w locator_to_server.pcap

       Explanation:
       -i eth0: Use the correct network interface (use ip a or ifconfig to list)
       host 192.168.1.50: Filter traffic from/to that server IP
       port 10334: Capture only the locator port
       -w locator_to_server.pcap: Save the output as a .pcap file for later analysis

   b. Run tcpdump on the Server 
       To capture traffic from the server side, run:
      sudo tcpdump -i eth0 host <locator-ip> and port 10334 -w server_to_locator.pcap

3. Analyze the Capture
You can open .pcap files using tools like:
   Wireshark (GUI)
   tcpdump CLI (tcpdump -r locator_to_server.pcap)