We have noticed that after some of our Secondary SpectroSERVERs are restarted, there are instances where processd does not start up. When we check further, we can see several processd core files located in the /var/crash directory.
When we run GDB on the processd core file we can see the following
gdb processd -c <core file name>
(gdb) bt
#0 CsHostAddress::ntoa (this=0x260) at CsHostAddr.cc:1075
#1 0x00000000004db42d in CsServerConnection::do_handshake (this=this@entry=0x3e57df8,
sock_type=sock_type@entry=CsFdList::VPAPI_ST, fd=fd@entry=7) at CsSrvConn.cc:500
#2 0x00000000004dba3d in CsServerConnection::check_register_of_fd (this=this@entry=0x3e57df8,
connection_fd=connection_fd@entry=5, listen_cif=<optimized out>) at CsSrvConn.cc:362
#3 0x00000000004dbb35 in CsServerConnection::retrieve_ticket (this=this@entry=0x3e57df8)
at CsSrvConn.cc:2087
#4 0x00000000004d41a4 in CsPTTServer::inner_processd_event_parser (this=0x3e57d50) at CsPTTServer.cc:852
#5 0x00000000004d49be in CsPTTServer::processd_event_loop (this=0x3e57d50) at CsPTTServer.cc:823
#6 0x00000000004d3853 in Processd::run (this=0x226fb20 <theDaemonObj>) at processd.cc:322
#7 0x000000000052b919 in UnixDaemonBase::start (this=0x226fb20 <theDaemonObj>, debug=<optimized out>,
argc=<optimized out>, argv=<optimized out>) at UnixDaemonBase.cc:988
#8 0x00000000004d2ba6 in UnixDaemonBase::main (this=0x226fb20 <theDaemonObj>, argc=<optimized out>,
argv=0x7ffdb74d3d28) at UnixDaemonBase.cc:297
#9 0x00007f8958495555 in build_trtable () from /lib64/libc.so.6
Release : 21.2.x / 22.2.x
Component : Spectrum Core / SpectroSERVER
Many old and unnecessary hostnames in the $SPECROOT/.hostrc file that would cause processd not to complete the host verification process upon processd startup.
Removed the unneeded hostfile names from the $SPECROOT/.hostrc file.
Now when we reboot the server we no longer see processd crashing.