Spectrum online backup run on the Primary SpectroSERVER not syncing the database to the Secondary SpectroSERVER

Products

• Spectrum
• Network Observability

Issue/Introduction

Online Backup (OLB) run on the Primary SpectroSERVER is not syncing the database to the Secondary SpectroSERVER. The RCPD.OUT file on the Primary SpectroSERVER system shows the following:

text

command: SEND
host: ####
file: /opt/Spectrum/SS-DB-Backup/db_20260915_0738.SSdb
rcpd: 0xcafe
procd: 0xfeeb
compress: 1

Nov 09 06:42:32 : Waiting for remote processd to startup rcpd...
Nov 09 06:45:02 : Retrying connection to remote rcpd...
Nov 09 06:45:27 : Unable to connect to remote rcpd. Exiting!
Nov 09 06:45:27 : Final status is -1

The Secondary SpectroSERVER may show: Waited for 3 minutes but did not accept a connection from the primary rcpd.

The Secondary SpectroSERVER  may have not created the RCPD.OUT file, indicating the rcpd process was not run.

 

Environment

DX NetOps Spectrum all currently supported releases

Cause

Communication is blocked between the Primary and Secondary SpectroSERVERs. This is typically caused by:

• A network firewall blocking the required ports.
• The OS-level firewall (such as firewalld or iptables on Linux) being enabled or reset to default rules.
• The processd service being stopped on the Secondary SpectroSERVER.

 

 

Resolution

To resolve the synchronization failure, perform the following steps:

1. Verify Process Status: Ensure that the processd service is running and active on the Secondary SpectroSERVER.
2. Check OS Firewalls: On Linux systems, check the status of firewalld. If it is active, ensure rules exist to allow Spectrum traffic or disable it if it was enabled in error.
   • Command to check: systemctl status firewalld
3. Ensure Port Availability: Verify the following ports are open bi-directionally between the Primary and Secondary systems:
   • 65259 (processd)
   • 61904 (processd)
   • 51966 (rcpd)
4. Test Connectivity: Use the ncat command from the Primary server to test communication to the Secondary server on the required ports:
   • nc -zv <Secondary_Hostname_or_IP> 51966
5. Review Host Files: Ensure the /etc/hosts (Linux) or C:\Windows\System32\drivers\etc\hosts (Windows) file on both servers contains correct entries for the peer server.

Additional Information

Verify connectivity from primary to secondary SS on these ports below:

• 65259 (0xfeeb) (processd)
  
  (on the primary SS run) nc -zv <secondary SS> 65259
  
  
• 61904 (0xf1d0) (processd)
  
  (on the primary SS run) nc -zv <secondary SS> 61904
  
  
• 51966 (0xcafe) (rcpd)
  
  (on the primary SS run)
  cd $SPECROOT/SS/
  ../SS-Tools/rcpd -recv -f .ft_save_file.SSdb
  
  rcpd will run for 3 minutes and it will end automatically. So run the next command while the rcpd is running on the secondary SS.
  
  (on the primary SS run) nc -zv <secondary SS> 51966

 

 

Please reference the "Communication Across Firewalls" section of the documentation for more information:

TechDocs : DX NetOps Spectrum 25.4 : Enable Communication Across Firewalls