OLB and MM stop working in Spectrum 23.3.12 and 23.3.13

Products

• DX NetOps
• CA Spectrum

Issue/Introduction

After upgrading to NetOps Spectrum 23.3.12 and/or 23.3.13 there are performance issues that may be seen such as:

Online Backup does not launch and no longer works
Maintenance schedules may not run or they may run later than they should
The NCM Network Configuration Manager Global Sync task fails to run
NCM Host Configuration captures using tftp fail

To try and determine what may be causing this, you can run the kill command with the "-USR1" flag.  This will not actually kill the process, it will dump a stack of the SpectroSERVER to the $SPECROOT/SS/.moot.trace file.

Obtain the pid of the SpectroSERVER process (ps -ef | grep SpectroSERVER).

Run kill -USR1 pid

For example:

kill -USR1 2212

Review the .moot.trace file and see if there are entries for IHHostConfig get_transfer_file_content:

IHHostConfig::get_transfer_file_content(CsModelHandle const&, char const*, CsError::CsError_e&, unsigned int) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::check_capture_transfer(CsModelHandle const&, CsError::CsError_e&, int&) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::get_captured_config(CsModelHandle const&, CsError::CsError_e&) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::capture_running_startup(CsModelHandle const&, CsError::CsError_e&) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::do_capture(CsModelHandle const&) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::handle_capture_request(CsModelHandle const&, CsAction const*) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]
IHHostConfig::trig_action(CsModelHandle const&, CsAction const*) [/opt/CA/Spectrum/SS/Modules/SA-SCMSS.so.1]

Cause

There is bug where the function hangs in the tftp transfer when using tftp for NCM.

Resolution

Edit the $SPECROOT/SS/.vnmrc and add the following parameter:

ncm_tftp_wait=1

Stop and restart the SpectroSERVER.