SSdb OnLineBackup Database Synchronization fails because of RCPD

Products

• Spectrum
• Network Observability

Issue/Introduction

I have tried all recipes found in the Knowledge base, but the OnLineBackup keeps failing with:

[root@<hostName>]# cat /usr/SPECTRUM/SS/RCPD.OUT
Apr 26 20:11:38 : rcpd started

 command:  SEND
    host:  spectrum
    file:  /usr/SS-DB-Backup/db_10.4.3_20220426_2010.SSdb
    rcpd:  0xcafe
   procd:  0xfeeb
compress:  1

Apr 26 20:11:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:12:08 : Waiting for remote processd to startup rcpd...
Apr 26 20:12:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:13:08 : Waiting for remote processd to startup rcpd...
Apr 26 20:13:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:14:08 : There is no remote rcpd running on spectrum.  Exiting

 

Also, applied the steps from the KB article below:

https://knowledge.broadcom.com/external/article?articleId=210301

Environment

Release : 22.2

Component : Spectrum Core / SpectroSERVER

Cause

Steps performed to investigate the issue:

a) Created the $SPECROOT/SS/.rcpdrc file and add the following line:

verbose=TRUE

The RCPD.OUT log file will be in verbose mode.

You can either remove the .rcpdrc file or set the verbose parameter to FALSE.

b) Enabled OnLineBackup in debug mode on the VNM model, under the Dynamic Debugging subview.

c) Enabled the processd in debug mode as the root user:

$ kill -trap <PID of processd>

d) Run the following command line on the secondary SS as the root user while the Database synchronization is in progress:

$ while :; do ps -efl | grep rcpd; sleep 1; done

$ netstat -anlp | grep 51966

e) Edited the $SPECROOT/lib/SDPM/processd_init.sh file and added the following lines on the secondary SS (before export PROCESSD_LCOPY=TRUE entry):

export PROCESSD_DEBUG=TRUE
export PROCESSD_DEBUG_LOGFILE=secondary_processd_OLB.log
export PROCESSD_LCOPY=TRUE

And cycled the processd process.

$ systemctl stop processd 

$ systemctl start processd 

f) Rerun the OnLineBackup to trigger the Database Synchronization task.

g) Review the $SPECROOT/lib/SDPM/secondary_processd_OLB.log file.

The required field <TICKETUSER> is missing from /usr/SPECTRUM/lib/SDPM/partslist/RCPD.idb install ticket.

h) The $SPECROOT//lib/SDPM/partslist/RCPD.idb was modified by the customer for troubleshooting.

# Processd Install Ticket for SPECTRUM Fault Tolerant Database Synchronization.
PARTNAME;RCPD;
APPNAME;SPECTRUM Fault Tolerant Database Synchronization;
WORKPATH;$SPECROOT/SS;
#LOGNAMEPATH;$WORKPATH/RCPD.OUT;
ADMINPRIVS;y;
AUTORESTART;Y;
AUTOBOOTSTART;Y;
#STATEBASED;N;
NUMPROCS;2; // one per SpectroSERVER
RETRYTIMEOUT;0; // seconds
TICKETUSER;spectrum;
RETRYMAX;0;     // retries
STARTPRIORITY;30;
SERVERPROCESS;N;
#ENV;<var>=<value>;
ARGV;$SPECROOT/SS-Tools/rcpd<CSEXE>; //

Resolution

Removed the changes made on the RCPD.idb file:

# Processd Install Ticket for SPECTRUM Fault Tolerant Database Synchronization.
PARTNAME;RCPD;
APPNAME;SPECTRUM Fault Tolerant Database Synchronization;
WORKPATH;$SPECROOT/SS;
#LOGNAMEPATH;$WORKPATH/RCPD.OUT;
ADMINPRIVS;y;
#AUTORESTART;n;
#AUTOBOOTSTART;n;
#STATEBASED;N;
NUMPROCS;2; // one per SpectroSERVER
RETRYTIMEOUT;0; // seconds
#TICKETUSER;spectrum;
RETRYMAX;0;     // retries
STARTPRIORITY;30;
SERVERPROCESS;N;
#ENV;<var>=<value>;
ARGV;$SPECROOT/SS-Tools/rcpd<CSEXE>; //

Cycled the processd process.

At this time the OnLineBackup successfully synchronized the SSdb on the secondary SS.