Spectrum Network Configuration Manager general failure

Products

• Spectrum
• Network Observability

Issue/Introduction

 

 Network Configuration Manager service is not working after upgrading DX NetOps Spectrum. When restarting all DXNetOps Spectrum services, it appears the "NCM" service (managed by processd) is running ok but it is not.

Network Configuration Manager (NCM) service is a standalone service task (ncmservice - CORBA resource name "scmservice") running besides the SpectroSERVER. When running NCM task the SpectroSERVER will establish communication with the NCM-service and the NCM-service is then processing/acting this task.

Used tools here are:
 ./bin/cmdC        - to check for  processd managed services
 ./bin/VBNS/nsutil   - to check for CORBA resource registration names and context
 netstat -an | grep 52161   - to check for socket 52161 status 
 netstat -an[p|b]      - to check for application output to see proess/task name per socket

Environment

This applies to all supported CA Spectrum platforms and releases.

Resolution

IP-stack/protocol reconfiguration solves this problem. In case IPv6 and IPv4 is enabled then ensure name-resolution for IPv6 is also fine (and may not point to "localhost" for hostname).

 

Additional Information

The NCM-service is running as "ncmservice" task and is using CORBA communication for the paired userSERVER<->ncmservice communication.
This is a TCP-session establishment from userSERVER to the ncmservice task. "ncmservice" task will open socket 52161 in LISTEN.
Still - we do not see an established TCP-session for userSERVER <-> ncmservice

[user@hostname VBNS]$ netstat -an | grep 52161
tcp    0   0 :::52161            :::*        LISTEN

[user@hostname VBNS]$ netstat -anp | grep 52161
tcp    0   0 :::52161            :::*        LISTEN      30532/ncmservice


Background here is the CORBA resource name registration and the naming-service - which expects to have all CORBA service registered under context of the "hostname".
This could be checked by tool ./bin/VBNS/nsutil

[user@hostname VBNS]$ ./nsutil list hostname  (output here is correct/expected)
Bindings in hostname
Object : SLMDomain
Object : LmtManagerDomain
Object : RTMDomain
Object : SpectrumICMP
Object : CsCLocServMapInt
Object : TelcoManagerDomain
Object : CsCEventDomain
Object : ScmService
Object : EventDispDomain
Object : CsCModelDomain
Object : SnmpServ
Object : ProcedureDomain
Object : OnlineMibImport
Object : CsCAlarmDomain

For this problem here the "ScmService" was not listed under context of hostname - but was listed under context of "localhost" - seeing:
[user@hostname VBNS]$ ./nsutil list localhost
Bindings in localhost
Object : ScmService

Interesting and confusing - the "ScmService" is the only CORBA resource allocation registered under "localhost" while all others are fine under context of "hostname".