GemFire: Best Practices for TCP/UDP
search cancel

GemFire: Best Practices for TCP/UDP

book

Article ID: 418433

calendar_today

Updated On:

Products

VMware Tanzu Gemfire VMware Tanzu Greenplum / Gemfire VMware Tanzu Data Suite VMware Tanzu Data Suite VMware Tanzu Data Intelligence

Issue/Introduction

GemFire supports the use of both TCP and UDP for communications. Depending on the size and nature of the system as well as the types of regions employed, either TCP or UDP may be more appropriate.

Resolution

TCP Communications

TCP (Transmission Control Protocol) provides reliable, ordered message delivery. GemFire uses TCP by default for inter-cache point-to-point messaging.

When TCP Is Preferable

  • Partitioned Data Systems using partitioned regions benefit from TCP’s reliability and performance; TCP handles large, frequent point-to-point messaging more efficiently than UDP.

  • Smaller Distributed Systems
    TCP’s OS-level reliability and speed often outperform UDP unicast in smaller clusters.

  • Unpredictable Network Loads
    TCP is more tolerant of congestion and network instability. While GemFire adds its own retransmission layer to UDP, it cannot fully compensate for heavy network spikes.

Note:
GemFire always uses TCP for member failure detection. During suspect-member checks, GemFire opens a TCP connection to verify liveness.

UDP Communications

UDP (User Datagram Protocol) is connectionless and lightweight but has limitations:

  • 64 KB message size limit (including headers)

  • Slower on congested networks

  • Less reliable (GemFire compensates with retransmissions)

When UDP Is Preferable

  • Replicated Data with Multicast
    If most members use the same replicated regions, UDP multicast efficiently distributes events.
    However, multicast means every process receives all events for that region—only suitable when all members care about the same data.

    Partitioned regions still use UDP unicast for most operations.

  • Large Distributed Systems
    In larger clusters, UDP’s lower overhead scales better. TCP’s extra threads and sockets add overhead as cluster size grows.

Note: to configure GemFire to use UDP for inter-cache point-to-point messaging set the following GemFire property: disable-tcp=true

TCP Settings

GemFire-Specific TCP Settings

Socket Buffer Size

It balance communication throughput with memory usage.

  • Buffers should be large enough to hold the largest object + key + ~100 bytes header. This prevents fragmentation and improves performance.

Guidelines

  • Peer-to-peer: same  socket-buffer-size across all members

  • Client/server: client pool size should match server settings

  • Servers: server-side buffer should match its clients

  • WAN (Gateway): sender and receiver buffer sizes must match

Note: If OS-level TCP buffer limits are smaller than GemFire’s requested value, the OS limit will override GemFire settings.

 

TCP KeepAlive

It prevents idle connections from being closed by firewalls or network equipment. Actual KeepAlive intervals are controlled by OS settings. By default, this system property is set to true.

 

TCP/IP Peer-to-Peer Handshake Timeouts

This property controls timeout for completing TCP handshakes between peers. Default is 59,000 ms which can be changed by using -Dp2p.handshakeTimeoutMs=75000 while starting the servers. For example, 

 
gfsh>start server --name=server1 --J=-Dp2p.handshakeTimeoutMs=75000

 

Linux TCP/IP Settings for GemFire

Recommended entries for /etc/sysctl.conf:

Setting Recommended Value Rationale
net.core.netdev_max_backlog 30000 (10GbE) / 8000 (1GbE) Queue packets when kernel is slower than NIC
net.core.wmem_max 64MB (10GbE) / 16MB (1GbE) Increase max socket write buffer
net.core.rmem_max 64MB (10GbE) / 16MB (1GbE) Increase max socket read buffer
net.ipv4.tcp_congestion_control htcp Avoid issues with bic/cubic on older kernels
net.ipv4.tcp_congestion_window 10 Default for 2.6.39+ kernels
net.ipv4.tcp_fin_timeout 10 Free socket resources faster
net.ipv4.tcp_keepalive_intvl 30 Reduce interval between probes
net.ipv4.tcp_keepalive_probes 5 Retry for ~2.5 minutes
net.ipv4.tcp_keepalive_time 600 Shorten idle timeout to 10 minutes
net.ipv4.tcp_low_latency 1 Prefer low latency
net.ipv4.tcp_max_orphans 16384 Limit unswappable orphan memory
net.ipv4.tcp_max_tw_buckets 1440000 Limit TIME_WAIT sockets
net.ipv4.tcp_no_metrics_save 1 Disable TCP metrics caching
net.ipv4.tcp_orphan_retries 0 Limit retries for orphaned sockets
net.ipv4.tcp_rfc1337 1 Protect from TIME_WAIT assassination
net.ipv4.tcp_rmem 10240 131072 33554432 Increase receive buffer autotuning limits
net.ipv4.tcp_wmem 10240 131072 33554432 Increase send buffer autotuning limits
net.ipv4.tcp_sack 1 Enable selective ACKs
net.ipv4.tcp_slow_start_after_idle 0 Avoid slow-start penalty
net.ipv4.tcp_syncookies 0 Disable SYN cookies (if protected cluster)
net.ipv4.tcp_timestamps 1 Enable timestamps
net.ipv4.tcp_tw_recycle 1 Faster recycling of TIME_WAIT (use with caution)
net.ipv4.tcp_tw_reuse 1 Safely reuse TIME_WAIT sockets
net.ipv4.tcp_window_scaling 1 Allow large TCP windows

In addition, increasing the size of transmit queue can also help TCP throughput. Add the following command to /etc/rc.local to accomplish this.

/sbin/ifconfig eth0 txqueuelen 10000

NOTE: substitute the appropriate adapter name for eth0 in the above example.

Powered by Wolken Software