More consideration on TCP port assignment
Article ID: 294209
Updated On:
Products
Issue/Introduction
There are several configuration parameters within GemFire used to specify various TCP ports: locator port, cache server port, membership port range, and so on. Generally, GemFire assigns TCP ports based on these configuration settings, however, there are others that are assigned based on OS or JVM behavior. This article discusses those OS/JVM managed TCP ports and how to prevent to compete in terms of TCP port assignment between GemFire and OS or JVM.
Environment
Resolution
Suppose you start a locator and a cache server configured with TCP port parameters set similar to the following:
locator's listen port = 55221 cache server's listen port = 43331 jmx-manager-port = 1099 jmx-manager-http-port = 8080 membership-port-range = 50000-50050 disable-tcp=true <-- eliminate standard TCP sockets for the purpose of this example
Checking TCP connections and listening ports for each member using the lsof command, shows additional (unspecified) TCP sockets similar to those indicated (by numeric comments) in the following example:
$ lsof -n -P -p 2999 | grep TCP <-- check locator process java 2999 gemfire 24u IPv6 21041 0t0 TCP 172.16.227.52:8080 (LISTEN) java 2999 gemfire 32u IPv6 20657 0t0 TCP *:55221 (LISTEN) java 2999 gemfire 37u IPv6 20668 0t0 TCP *:12630 (LISTEN) <-- (1) java 2999 gemfire 40u IPv6 20694 0t0 TCP 172.16.227.52:50037 (LISTEN) java 2999 gemfire 43u IPv6 65651 0t0 TCP 172.16.227.52:50037->172.16.227.52:8798 (ESTABLISHED) <-- (2) java 2999 gemfire 44r IPv6 21484 0t0 TCP 172.16.227.52:37123->172.16.227.52:50026 (ESTABLISHED) <-- (2) java 2999 gemfire 47r IPv6 21254 0t0 TCP *:1099 (LISTEN) $ lsof -n -P -p 3293 | grep TCP <-- check cache server process java 3293 gemfire 34w IPv6 21454 0t0 TCP *:36092 (LISTEN) <-- (1) java 3293 gemfire 36u IPv6 21456 0t0 TCP 172.16.227.52:50026 (LISTEN) java 3293 gemfire 38u IPv6 65650 0t0 TCP 172.16.227.52:8798->172.16.227.52:50037 (ESTABLISHED) <-- (2) java 3293 gemfire 39r IPv6 21485 0t0 TCP 172.16.227.52:50026->172.16.227.52:37123 (ESTABLISHED) <-- (2) java 3293 gemfire 44u IPv6 21525 0t0 TCP 172.16.227.52:43331 (LISTEN)
The listening ports indicated by a (1) are opened internally by the JVM when JMX is enabled for the RMI registry and are distinct from the jmx-manager-port. These ports are assigned from the ephemeral port range, which is configured as an OS/kernel parameter.
The connections, (2), are for the JGroups FD_SOCK listener/pinger for the purpose of failure detection of GemFire members. The server side ports for FD_SOCK listener/pinger are selected based on the membership-port-range setting while the client side ports are selected from ephemeral ports.
Sockets assigned ephemeral ports may be assigned any open port within the configured ephemeral range and could occupy otherwise reserved ports that are not currently (at the time of socket creation) being used. GemFire can use many ports so it is recommended that the range of ephemeral ports and those specified in GemFire's configuration not overlap.
Generally, it is easiest to configure GemFire's ports outside the ephemeral range. However, if necessary, you might also modify the range of ephemeral ports using an OS specific method. In the case of Red Hat Linux, you can modify this range by setting the net.ipv4.ip_local_port_range kernel parameter to an appropriate range in the /etc/sysctl.conf file and rebooting the machine. For example, adding
net.ipv4.ip_local_port_range = 56000 60000
to the /etc/sysctl.conf file will set the range of ephemeral ports to between 55000 and 60000.
Additional Information
References
GemFire docs: Ephemeral TCP Port Limits
This document mentions on how to modify the range of ephemeral ports for Windows platform.Feedback
