After upgrading to Spectrum 9.4.4 we have noticed our scheduled AutoDiscoveries fail to model. The modeling process is always locked out due to another process that is running.
With the release of Spectrum 9.4.4 we provided some additional functionality that allows us to place all background discoveries in the "Pending Rediscovery" queue. However, AutoDiscoveries are not consider background discoveries, and therefore they are never added to this queue. The TopologyWrkSpc model keeps track of the AutoDiscovery modeling requests, and will process them once the "Pending Discovery" queue is empty.
If your network is always changing, or you have a flood of traps or reconfigurations, this can cause this queue to never empty, the TopologyWrkSpc will not be able to process any modeling requests because the "Pending Discovery" queue will keep the work space locked.
To determine how deep your "Pending Discovery" queue is backed up:
1) Enable the AutoDiscovery Debug on the VNM model:
a) Open a OneClick client
b) Locate the VNM model in OneClick
c) In the Component Detail pane - Information tab, expand the AutoDiscovery Control subview
d) Expand the Debug Options subview
e) Set Debug AutoDiscovery to On. This will create an ADiscDebug_<date>_<time> file in the $SPECROOT/SS directory, which will contain the discovery activities for the VNM.
2) Open a remote connection to the SpectroSERVER running the discoveries.
3) Open a command line, or “bash -login” shell for Windows.
4) Change directories to $SPECROOT/vnmsh.
5) Open a CLI connection by running ./connect
6) Find the model handle of the "AutoDiscovery" model associated with the "TopologyWrkSpc" Model Type:
a) Run: ./show models | grep -i Topo.
b) This will return 4 columns of data, where:
“Model Handle” “Model Name” “Model Type Handle” “Model Type Name”
Which should look something that like:
0x50002c AutoDiscovery 0x25e0001 TopologyWrkSpc
0x500005 TopOrg 0x102cf Top_Org
c) Make note of the model handle of the AutoDiscovery model.
7) Run the following update action to print out the contents of the “Pending Discovery” queue: ./update action=0x10601 mh=<Model Handle of the AutoDiscovery model>
For example: ./update action=0x10601 mh=0x50002c
For Spectrum 9.4.4 the output of the Pending Discovery queue will be found in the $SPECROOT/SS/VNM.OUT file.
For Spectrum 10.x the output of the Pending Discovery queue will be found in the $SPECROOT/SS/ADiscDebug_<date>_<time> log.
If you find the “Pending Discovery” queue is backed up, you may want to clear the queue by running ./update action=0x10602 mh=<Model handle of the AutoDiscovery model>
Run update action 0x10601 again to check the contents of the queue.
If the queue is empty, you will need to abort the currently running discovery by:
1) Locating the VNM model in OneClick.
2) Select the Information Tab in the Component Detail pane
3) Expand the “AutoDiscovery Control” subview and select the “Abort Discovery” button.
Check the contents of the “Pending Discovery” queue once again to ensure it is still empty.
Due to the dynamic complexity of our network, we must have Spectrum configured to reconfigure/rediscover devices anytime a link has changed, in order to help keep the network topology as accurate as possible. We also rely in trap based discoveries to help us automatically discover any "one-off" devices that may have been missed during the scheduled discoveries. However, after upgrading to Spectrum 9.4.4 we noticed the AutoDiscoveries are no longer modeling due to the modeling being locked by another process.
If you find the "Pending Discovery" queue is filling up faster than you can clear it, and the AutoDiscovery modeling session continues to be locked out, you will want to apply Spectrum_09.04.04.D140 (Spectrum 9.4.4), Spectrum_10.02.00.PTF_10.2.021(Spectrum 10.2), or Spectrum_10.02.01.PTF_10.2.111 (Spectrum 10.2.1) to your SpectroSERVER.
This patch provides an option to disable the "Pending Discovery" queue feature, and will allow the AutoDiscovery modeling to take place once the currently running background discovery completes.
To install the patch:
Post Installation Steps:
After installing the patch you will need to add the following line to the $SPECROOT/SS/.vnmrc file to disable the "Pending Discovery" queue:
This change will be read in and take affect once the SpectroSERVER is restarted.
Sustaining Engineering has provided a more permanent solution, that will provide some prioritization around these requests, that way the queuing will not have to be disabled. The prioritization has been included in CA Spectrum 10.3 and above.