Performance Management Copy Cluster failing with connection timeout

Products

• CA Performance Management
• Network Observability

Issue/Introduction

Copy cluster is failing with following error:

Mon Oct 24 02:00:10 EDT 2022: Starting copy cluster script
Configured backup directories in Mapping are ignored for copy cluster.
Error: Timed out connecting to rsync daemon on host <destination_Host> port 50000: [Errno 111] Connection refused.
Copycluster FAILED.
Mon Oct 24 02:01:21 EDT 2022: copy cluster finished

DX NetOps Performance Management Data Repository copycluster fails due to connection refused for port 50000.

Environment

All supported DX NetOps Performance Management releases

Cause

Different causes have been seen for this error.

• Firewall blocking rsync port between hosts
• The selinux configuration is set to ENFORCING
• Another tool is using the rsync port configured
  
  • To determine if another tool is using port 50000:
    • Run the following as the root user and note the PID.
      • netstat -anp | grep 50000 | grep LISTEN
    • The output might look similar to:
      • tcp        0      0 127.0.0.1:50000         0.0.0.0:*               LISTEN      2330/oneagentwatchd
    • Take the PID, 2330 in the example output and run this to identify the process.
      • ps -ef | grep PID
• A failed copycluster has left a hung rsync process tied to port 50000
  • When copycluster runs it should start rsync on the source and destination, and close it when completed.
  • There should be no rsync processes remaining after a successful copycluster.

Resolution

• Firewall blocks on the systems or network?
  • Ensure firewalld is down on both systems
  • Ensure network firewalls between systems allow port 50000 traffic between hosts
• Reconfigure selinux so it is (ideal fix) disabled, or (less ideal but functional) set to PERMISSIVE.
• Another tool is using port 50000?
  • Shut down or disable the tool.
  • Not able to disable the tool? Configure an alternative port in the copycluster *.ini configuration file.
    • Ensure the newly defined port is open from source to destination.
    • Many found success simply using port 60000 as an alternative
    • To set another port to use in the ini configuration file for your copycluster add an entry under the [Transmission] header like this: port_rsync = 60000
• Kill any running rsync services on all nodes and restart the copycluster.
  • Easy method is using "kill -9 PID" where PID is the PID of the leftover rsync process.