SSdb OnLineBackup Database Synchronization fails because of RCPD
Article ID: 241046
Updated On:
Products
Issue/Introduction
I have tried all recipes found in the Knowledge base, but the OnLineBackup keeps failing with:
[[email protected]]# cat /usr/SPECTRUM/SS/RCPD.OUT
Apr 26 20:11:38 : rcpd started
command: SEND
host: spectrum-2
file: /usr/SS-DB-Backup/db_10.4.3_20220426_2010.SSdb
rcpd: 0xcafe
procd: 0xfeeb
compress: 1
Apr 26 20:11:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:12:08 : Waiting for remote processd to startup rcpd...
Apr 26 20:12:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:13:08 : Waiting for remote processd to startup rcpd...
Apr 26 20:13:38 : Waiting for remote processd to startup rcpd...
Apr 26 20:14:08 : There is no remote rcpd running on spectrum-2. Exiting
Also, applied the steps from the KB article below:
https://knowledge.broadcom.com/external/article?articleId=210301
Environment
Release : 20.2
Component : Spectrum Core / SpectroSERVER
Cause
Steps performed to investigate the issue:
a) Created the $SPECROOT/SS/.rcpdrc file and add the following line:
verbose=TRUE
The RCPD.OUT log file will be in verbose mode.
You can either remove the .rcpdrc file or set the verbose parameter to FALSE.
b) Enabled OnLineBackup in debug mode on the VNM model, under the Dynamic Debugging subview.
c) Enabled the processd in debug mode as the root user:
$ kill -trap <PID of processd>
d) Run the following command line on the secondary SS as the root user while the Database synchronization is in progress:
$ while :; do ps -efl | grep rcpd; sleep 1; done
$ netstat -anlp | grep 51966
e) Edited the $SPECROOT/lib/SDPM/processd_init.sh file and added the following lines on the secondary SS (before export PROCESSD_LCOPY=TRUE entry):
export PROCESSD_DEBUG=TRUE
export PROCESSD_DEBUG_LOGFILE=secondary_processd_OLB.log
export PROCESSD_LCOPY=TRUE
And cycled the processd process.
$ systemctl stop processd
$ systemctl start processd
f) Rerun the OnLineBackup to trigger the Database Synchronization task.
g) Review the $SPECROOT/lib/SDPM/secondary_processd_OLB.log file.
The required field <TICKETUSER> is missing from /usr/SPECTRUM/lib/SDPM/partslist/RCPD.idb install ticket.
h) The $SPECROOT//lib/SDPM/partslist/RCPD.idb was modified by the customer for troubleshooting.
# Processd Install Ticket for SPECTRUM Fault Tolerant Database Synchronization.
PARTNAME;RCPD;
APPNAME;SPECTRUM Fault Tolerant Database Synchronization;
WORKPATH;$SPECROOT/SS;
#LOGNAMEPATH;$WORKPATH/RCPD.OUT;
ADMINPRIVS;y;
AUTORESTART;Y;
AUTOBOOTSTART;Y;
#STATEBASED;N;
NUMPROCS;2; // one per SpectroSERVER
RETRYTIMEOUT;0; // seconds
TICKETUSER;spectrum;
RETRYMAX;0; // retries
STARTPRIORITY;30;
SERVERPROCESS;N;
#ENV;<var>=<value>;
ARGV;$SPECROOT/SS-Tools/rcpd<CSEXE>; //
Resolution
Removed the changes made on the RCPD.idb file:
# Processd Install Ticket for SPECTRUM Fault Tolerant Database Synchronization.
PARTNAME;RCPD;
APPNAME;SPECTRUM Fault Tolerant Database Synchronization;
WORKPATH;$SPECROOT/SS;
#LOGNAMEPATH;$WORKPATH/RCPD.OUT;
ADMINPRIVS;y;
#AUTORESTART;n;
#AUTOBOOTSTART;n;
#STATEBASED;N;
NUMPROCS;2; // one per SpectroSERVER
RETRYTIMEOUT;0; // seconds
#TICKETUSER;spectrum;
RETRYMAX;0; // retries
STARTPRIORITY;30;
SERVERPROCESS;N;
#ENV;<var>=<value>;
ARGV;$SPECROOT/SS-Tools/rcpd<CSEXE>; //
Cycled the processd process.
At this time the OnLineBackup successfully synchronized the SSdb on the secondary SS.
Feedback
