Spectrum: PRIMARY SDC DISCONNECTED WITH NCM SERVICE

Products

• CA Spectrum

Issue/Introduction

After upgrading the SDCs, I get the following alarm in Spectrum:

"PRIMARY SDC DISCONNECTED WITH NCM SERVICE"

Both the SDC and the NCM process are running but the SDC doesn't connect to the NCM service. This is how it looks like on the Primary SDC:

The NCMSERV.OUT looks the same for both servers and doesn't show any errors. The SDC service on the Primary has already been restarted.

Environment

Release : 21.2

Component : Spectrum Core / SpectroSERVER

Cause

Misconfigured etc_hosts file.

Resolution

After enabling SDC debug we saw the following entry.

Wed Oct 27 16:17:04 2021: INFO: NCMServiceClient::GetStatus() health_check= Not OK!! , error_code = 4

Based on this we checked the SDCs /etc/hosts file and there were missing / wrong entries.  After correcting them the SDC and ncmservice processes are now connected and the according Spectrum alarm was cleared.

Additional Information

To enable debug:

Debug can be enabled on both the SDM and SDC by adding the option –loglevel to the sdm.config or sdc.config.

The levels of debug are:
-loglevel error          Only error messages are logged (least verbose)
-loglevel warning    Warning messages and errors are logged into sdmlog.log  (recommended and default setting)
-loglevel info    Informational, warning and error messages are logged into sdmlog.log
-loglevel debug   Verbose debug messages are logged into the sdmlog.log

If –loglevel debug is used then –maxlogsize 50 should also be used to keep the log file sizes down default is 10.

For most issues loglevel warning should provide sufficient amount of information in troubleshooting SDM / SDC connectivity issues.  If there are no messages in the sdmlog.log then everything is working correctly.
Loglevel  debug should not be used unless directed by Engineering as more often it contains too much information and it can be difficult to find where the real problem lies.
  * Remember that there can be only one line in the config files so never hit carriage return or the options will be ignored *