Troubleshoot: GPText address already in use segment error

Products

• VMware Tanzu Greenplum

Issue/Introduction

This article focuses on how to troubleshoot GPText startup or segment recovery issues related to ports being already in use. These topics apply to all versions of GPText on RHEL 6.x

This article focuses on how to troubleshoot GPText startup or segment recovery issues related to ports being already in use.

Symptoms:

gptext-start or gptext-recover will fail and in the Solr startup.log shows an error similar to the following:

java bind exception - address already in use
FAILED selectchannelconnector@0.0.0.0:1254: java.net.bindexception
oejuc.abstractlifecycle:FAILED org.eclipse.jetty.server.server@36f3bc82

Environment

OS: RHEL 6.x

Cause

As the error message suggests, the cause of this issue is that a GPText Solr instance cannot bind to the specified port because some other process is already using it.

Checking the local port range in the system might give an output similar to:

net.ipv4.ip_local_port_range = 1025 65535

This would mean that ports can be allocated by the OS for TCP and UDP traffic in a range from 1025 to 65535.

By default, GPText normally uses ports 11xx and 12xx. It is somewhat common that another process binds to one of these ports since these port numbers fall inside the system port range 1025 < 11xx < 12xx < 65535.

Resolution

GPText ports should be out of the range set by net.ipv4.ip_local_port_range. Otherwise, there is a risk of port conflict issues.

To set the correct range, configure the net.ipv4.ip_local_port_range parameter:
 

1. Stop GPText with the command: gptext-stop

2. Stop GPDB with the command: gpstop -af

3. As root, apply the following changes to all segment servers:

a. Change the /etc/sysctl.conf parameter net.ipv4.ip_local_port_range to a range not overlapping with GPText ports. For example, net.ipv4.ip_local_port_range = 1300 65535.
b. Reload sysctl.conf: sysctl -p
c. Restart network service: /etc/rc.d/init.d/network restart

4. Start GPDB with the command: gpstart -a

5. Start GPText with the command: gptext-start

Note: To retrieve tge current GPText configured ports, run gptext-state when GPText is up, or get the information from the jetty.conf files in the segments. GPText ports are specified during the GPText installation.

Checklist:
Confirm the following: 

• Check which process is currently using the same specific port.
• Check if solr.xml and jetty.conf files are correct in the segments.
• Check the local port range in the Linux system using the following command: sysctl -a | grep net.ipv4.ip_local_port_range