After upgrading the SDCs, I get the following alarm in Spectrum:
"PRIMARY SDC DISCONNECTED WITH NCM SERVICE"
Both the SDC and the NCM process are running but the SDC doesn't connect to the NCM service. This is how it looks like on the Primary SDC:
The NCMSERV.OUT looks the same for both servers and doesn't show any errors. The SDC service on the Primary has already been restarted.
Misconfigured etc_hosts file.
Release : 21.2
Component : Spectrum Core / SpectroSERVER
After enabling SDC debug we saw the following entry.
Wed Oct 27 16:17:04 2021: INFO: NCMServiceClient::GetStatus() health_check= Not OK!! , error_code = 4
Based on this we checked the SDCs /etc/hosts file and there were missing / wrong entries. After correcting them the SDC and ncmservice processes are now connected and the according Spectrum alarm was cleared.
To enable debug:
Debug can be enabled on both the SDM and SDC by adding the option –loglevel to the sdm.config or sdc.config.
The levels of debug are:
-loglevel error Only error messages are logged (least verbose)
-loglevel warning Warning messages and errors are logged into sdmlog.log (recommended and default setting)
-loglevel info Informational, warning and error messages are logged into sdmlog.log
-loglevel debug Verbose debug messages are logged into the sdmlog.log
If –loglevel debug is used then –maxlogsize 50 should also be used to keep the log file sizes down default is 10.
For most issues loglevel warning should provide sufficient amount of information in troubleshooting SDM / SDC connectivity issues. If there are no messages in the sdmlog.log then everything is working correctly.
Loglevel debug should not be used unless directed by Engineering as more often it contains too much information and it can be difficult to find where the real problem lies.
* Remember that there can be only one line in the config files so never hit carriage return or the options will be ignored *