Juniper routers chassis reconfiguring causing SpectroSERVER cpu problems and duplicate chassis modules in CA Spectrum

Products

• CA Spectrum

Issue/Introduction

Problem:

In CA Spectrum 10.1.0, 10.1.1, and 10.1.2, Juniper models may reconfigure every minute which can cause SpectroSERVER cpu problems and cause Spectrum to create duplicate chassis modules (which will lead to memory consumption).  When you upgrade to 10.2 or 10.3, the SS may consume 100% cpu and not show in the OneClick client.

 

If your Juniper device is modeled using the GnSNMPDev modeltype the following events are seen approximately once a minute on the model and duplicate chassis module models are created in the Chassis Manager view:

 

Apr 5, 2016 2:38:32 PM EDTSim26246GnSNMPDev (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001 

Apr 5, 2016 2:37:30 PM EDTSim26246GnSNMPDev (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001 

Apr 5, 2016 2:36:28 PM EDTSim26246GnSNMPDev (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001 

 

If your Juniper device is modeled using the JuniperJUNOSRtr modeltype the following events are seen approximately once a minute on the model but the duplication of modules does not happen:

 

Apr 5, 2016 2:38:32 PM EDTSim26246JuniperJUNOSRtr (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001 

Apr 5, 2016 2:37:30 PM EDTSim26246JuniperJUNOSRtr (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001 

Apr 5, 2016 2:36:28 PM EDTSim26246JuniperJUNOSRtr (name - Sim26246): The Chassis Manager has performed a reconfiguration of the chassis modeling. This is an attempt to maintain the accuracy of the modeling as changes to the actual device are detected. This may result in new board and port models being destroyed.System0x3d0001

 

 

SpectroSERVER performance dump files (They will be in the <SPECROOT>/SS/support directory) will show similar functions:

 #5 0x00007f6214a41c22 in CsALogHub::create_new_boards(hub_mf&, int&, unsigned int*, int, CsVPList*, CsOIDValueList*) () from /opt/SPECTRUM/lib/../SS/Modules/libgnsmpss.so.1

#6 0x00007f6214a423fb in CsALogHub::reconfigure(hub_mf&, CsAHub*, unsigned int*, unsigned int) () from /opt/SPECTRUM/lib/../SS/Modules/libgnsmpss.so.1

#7 0x00007f6214a4527d in CsIHChasMgr::compare_hub_config(CsModelHandle const&, CsAttrValList*, CsIHChasMgr::scratch*, unsigned int*, unsigned int) () from /opt/SPECTRUM/lib/../SS/Modules/libgnsmpss.so.1

#8 0x00007f6214a46d48 in CsIHChasMgr::trig_action(CsModelHandle const&, CsAction const*) () from /opt/SPECTRUM/lib/../SS/Modules/libgnsmpss.so.1

 

 

 

 

 

 

 

 

Environment

Release: SDBSFO99000-10.2-Spectrum-Device Based Suite-Server FOC
Component:

Cause

The problem is the CA Spectrum code was not accounting for inner modules on the Juniper Chassis configuration.  This would cause CA Spectrum to trigger a reconfigure of the chassis modeling.

Resolution

 

 

For CA Spectrum releases 10.1.0, 10.1.1, and 10.1.2 CA has created program temporary fixes (PTF).  

The PTF fixes are:

 

10.1.0 - Spectrum_10.01.00.PTF_10.1.025

10.1.1 - Spectrum_10.01.01.PTF_10.1.109

10.1.2 - Spectrum_10.01.02.PTF_10.1.205

 

The release notes show the patch is for the GnSNMPDev modeltype, however it is also for the JuniperJUNOSRtr modeltype:

 

Symptom: On the GnSNMPDev model, you will see that duplicate modules are created. 

 

Resolution: On the GnSNMPDev model, you will not see that duplicate modules are created. 

 

Please open a case with CA Support to obtain the PTF patch fix that you need for the version of CA Spectrum you are running.  CA Support will also need to provide you a copy of SSdebug to delete the duplicate modules after install of the patch. 

 

 

The fix is included in CA Spectrum 10.2 to stop the reconfiguration and creation of duplicate modules.  However if you experienced this problem in 10.x and did not know it, when you upgrade to 10.2 the SS cpu may be spiked at 100%.  If stack dumps of the SS show the entries above (#5, #6, #7) then you need to remove the duplicate JuniperSlot modules:

 

1.  Open a case with CA Support to obtain the SSdebug executable.

2.  Put the SSdebug executable in the SS directory on the SS in the <SPECROOT>/SS directory (change the permissions to be executable if need be).

3.  With the SpectroSERVER still running, execute the following command in CLI:

 

cd <SPECROOT>/vnmsh

 

connect

./show models mth=0x3b10033 | awk '{print "delete " $1}' > JuniperSlot.play 

4.  Move the JuniperSlot.play file to the SS directory.

5.  Save your SSdb.

6.  Stop the SpectroSERVER.  If it does not stop, you will need to crash the SS and reload your SSdb from either step #4 or from a previous SSdb savefile.

7.  Navigate to the SS directory in a bash shell.

8.  Execute the following command to mark the JuniperSlot modules for deletion:

./SSdebug -database -play JuniperSlot.play juniper.OUT

9.  Start the SpectroSERVER

10.  The VNM.OUT will note that the duplicate modules have been marked for deletion.

11.  The SS cpu will spike to 100% while it deletes the models.  This is normal and could take many hours (we have typically found that it takes approximately 1 hour for every 75000 models).

12.  Highlight all Juniper devices, right click and select the Reconfiguration -> Reconfigure Model.

 

 NOTE: After the reconfiguration is complete, you may see multiple module models with the same model name re-appear. If so, check the value of the jnxContentsDescr attribute on these models. If they are different, they are not duplicate module models. They are separate modules listed in the mib but have the same module name.

Additional Information

If you are not able to Start the SpectroSERVER, you will need to use SSdebug and run the following commands:

1. Reload your SpectroSERVER database 

2.  Place the SSdebug executable in the SS directory and provide execute permissions (linux)

3. in the SS directory, run:
./SSdebug -database -models | grep JuniperSlot | awk '{print "delete " $2}' > JuniperSlot.play 

4. Then run:
./SSdebug -database -play JuniperSlot.play juniper.OUT 

5. Then save the SSdb from the SCP 

6. Start the SpectroSERVER 
 

The VNM.OUT will note that the duplicate modules have been marked for deletion.

7.  The SS cpu will spike to 100% while it deletes the models.  This is normal and could take many hours (we have typically found that it takes approximately 1 hour for every 75000 models).

8.  Highlight all Juniper devices, right click and select the Reconfiguration -> Reconfigure Model.