Error: gpcopy rpc listen failed bind: address already in use

Products

• VMware Tanzu Greenplum
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

When running the gpcopy utility, the process fails with the following error even if a specific port is provided via the --rpc-port flag:

gpcopy rpc listen failed listen tcp 0.0.0.0:####: bind: address already in use

Users may also observe a warning indicating that the utility has switched to 'edge mode' despite being executed on the coordinator node:

[WARNING]:-It is recommended to run gpcopy on the coordinator node of the source gpdb cluster. If not on the recommended node, some operations will be run through the executable external table.

Environment

• VMware Tanzu Greenplum 6.x
• gpcopy utility 2.7.0 and higher

Cause

This issue is typically caused by one of two factors:

1. Port Conflict: Another gpcopy process or system service is already using the specified RPC port. Starting with gpcopy 2.7.0, the utility binds to a port (default 7667) for job management and monitoring. If the port is occupied, the bind fails.
2. Hostname Mismatch: gpcopy determines if it is running on the coordinator by comparing the local hostname to the hostname recorded in gp_segment_configuration (where content = -1). If these do not match, gpcopy defaults to edge mode, which can lead to unexpected port binding attempts or permission issues.

Resolution

To resolve the port conflict and ensure gpcopy recognizes the coordinator node, follow these steps:

1. Identify the process currently occupying the port using lsof or netstat:

   lsof -i :<port>
   
   or
   
   netstat -apn | grep <port>

2. If multiple gpcopy instances must run concurrently, assign a unique, available port to each instance using the --rpc-port flag:

   gpcopy --source-host <host> --dest-host <dest> --rpc-port 10001 --truncate

Also, to resolve the run gpcopy on the coordinator node error:

1. Verify the coordinator hostname in the database configuration:

   SELECT hostname FROM gp_segment_configuration WHERE content = -1 AND role = 'p';

2. Compare the result with the system hostname:

   hostname

3. If the hostnames do not match, update the system hostname or the database configuration to ensure they are identical. This prevents gpcopy from switching to edge mode when run on the coordinator. 

Note: The issue is fixed in gpcopy 2.9.5 now, we can also upgrade the gpcopy to avoid this issue.

Additional Information

For more details on managing parallel jobs, refer to the Tanzu Greenplum Data Copy Utility Reference.

To be updated on fix status for this issue, please subscribe to this article (see How to subscribe to a Knowledge Base Article).

If you need to speak with a customer representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.