Devices not modeling after auto-discovery runs as the modeling is locked by another process

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

We have noticed some of our scheduled AutoDiscoveries fail to model. The modeling process is always locked out due to another process that is running. 

Modeling is locked by another process.  Waiting for other process to finish...

Environment

Due to the dynamic complexity of our network, we must have Spectrum configured to reconfigure/rediscover devices anytime a link has changed, in order to help keep the network topology as accurate as possible. We also rely in trap based discoveries to help us automatically discover any "one-off" devices that may have been missed during the scheduled discoveries. We noticed the AutoDiscoveries are no longer modeling due to the modeling being locked by another process.

Cause

Spectrum will place all background discoveries in the "Pending Rediscovery" queue. However, AutoDiscoveries are not consider background discoveries, and therefore they are never added to this queue. The TopologyWrkSpc model keeps track of the AutoDiscovery modeling requests, and will process them once the "Pending Discovery" queue is empty. 

 

If your network is always changing, or you have a flood of traps or reconfigurations, this can cause this queue to never empty, the TopologyWrkSpc will not be able to process any modeling requests because the "Pending Discovery" queue will keep the work space locked.

 

To determine how deep your "Pending Discovery" queue is backed up:

1) Enable the AutoDiscovery Debug on the VNM model:

a) Open a OneClick client

b) Locate the VNM model in OneClick

c) In the Component Detail pane - Information tab, expand the AutoDiscovery Control subview

d) Expand the Debug Options subview

e) Set Debug AutoDiscovery to On. This will create an ADiscDebug_<date>_<time> file in the $SPECROOT/SS directory, which will contain the discovery activities for the VNM. 

2)  Open a remote connection to the SpectroSERVER running the discoveries.

3)   Open a command line, or “bash -login” shell for Windows.

4)   Change directories to $SPECROOT/vnmsh.

5)   Open a CLI connection by running ./connect

6)   Find the model handle of the "AutoDiscovery" model associated with the "TopologyWrkSpc" Model Type:

a)    Run: ./show models | grep -i Topo.

b)   This will return 4 columns of data, where:

“Model Handle”       “Model Name”         “Model Type Handle”         “Model Type Name”

 

Which should look something that like:

0x50002c    AutoDiscovery                  0x25e0001   TopologyWrkSpc

0x500005    TopOrg                            0x102cf     Top_Org

 

c)    Make note of the model handle of the AutoDiscovery model.

7)   Run the following update action to print out the contents of the “Pending Discovery” queue: ./update action=0x10601 mh=<Model Handle of the AutoDiscovery model>

     

     For example: ./update action=0x10601 mh=0x50002c

 

  For Spectrum 10.x and 22.x and above, the output of the Pending Discovery queue will be found in the $SPECROOT/SS/ADiscDebug_<date>_<time> log.   

 

If you find the “Pending Discovery” queue is backed up, you may want to clear the queue by running ./update action=0x10602 mh=<Model handle of the AutoDiscovery model>

Run update action 0x10601 again to check the contents of the queue.

If the queue is empty, you will need to abort the currently running discovery by:

1)   Locating the VNM model in OneClick.

2)   Select the Information Tab in the Component Detail pane

3)   Expand the “AutoDiscovery Control” subview and select the “Abort Discovery” button.

Check the contents of the “Pending Discovery” queue once again to ensure it is still empty.

Resolution

To prevent the discoveries from queuing so they will be dropped add the following line to the $SPECROOT/SS/.vnmrc file to disable the "Pending Discovery" queue:

ALLOW_DISCOVERY_QUEUING=FALSE

This change will be read in and take affect once the SpectroSERVER is restarted.