Spectrum Stale Connections don't change even though Auto_Discovery debug shows new port connection

Products

• Network Observability
• Spectrum

Issue/Introduction

Many devices show stale connections even though the ports' connectivity has been updated.  For example,  a new connection has been created in the ADisc_debug:

svi 19 18:13:03   Replace Network for Port CatalystSwitch_Gi0/1(ifType=6) having potential connections :1
svi 19 18:13:03     Port CatalystSwitch_Gi0/1(ifType=6) <--> Port System_GigabitEthernet0/0/0/1(ifType=6) 
Connection  (Cisco CDP) was locked: Port System_GigabitEthernet0/0/0/1(ifType=6) <--> Port CatalystSwitch_Gi0/1(ifType=6) 

But CatalystSwitch_Gi0/1 remains connected (and stale) with another port - System_GigabitEthernet0/0/3

In this situation, the following circumstances are observed:

1. Spectrum creates connections automatically. There is no manual intervention.
   
   
2. "Automatically Reconfigure Interface" is set to Yes. Rediscovered, reconfigured but there is no impact. Stale connections remain.
   
   
3. Tried OC logout/login and recycle of services but none helped to resolve the issue.
   
   
4. Issue affects multiple devices and has been ongoing for some time.

The LockConnection attribute id 0x129f1 on the interface model(s) at both sides has been set to unlocked and reconfiguration done on the devices. However, the same issue remains even after unlocking the interface before reconfiguring it. It remains stale.

Environment

DX NetOps Spectrum all currently supported releases

Cause

If the connection is locked and the same port is linked with any other connection, spectrum will not show new connections since it's a manual process due to locking.

In OneClick, you can preserve a resolved connection between two modeled devices by locking that connection. When you lock a connection, Subsequent Discovery does not delete it even if it is not found during that discovery. This is working as designed, see the following for further details: 

TechDocs : DX NetOps Spectrum 24.3 : Lock and Unlock Resolved Connections

 

Resolution

The script attached to this KB (reconfigUtility_updated.sh) has been provided by Broadcom Level 2 Sustaining Engineering and identifies the manual connections on the port models, then triggers the reconfiguration actions on the device models. This then allows reconfiguration to eliminate the stale connections.

This script has to be executed manually after placing the shell script under $SPECROOT/vnmsh folder. Then you run the following commands to give it user and execute permissions:

chown <SPECTRUM install user>:<Spectrum install group> $SPECROOT/vnmsh/reconfigUtility_updated.sh
chmod +x $SPECROOT/vnmsh/reconfigUtility_updated.sh

So for example, at the command line, if spectrum is the name of the install owner and tools is the name of the group it belongs to and /usr/Spectrum/ is SPECROOT, then copy the script to :

cd /usr/Spectrum/vnmsh

Then run:

chown spectrum:tools  reconfigUtility_updated.sh
chmod +x reconfigUtility_updated.sh

Then run the script as per instructions.

./reconfigUtility_updated.sh