10.2.2 and above SpectroSERVER model activation taking many hours

Products

• CA Spectrum

Issue/Introduction

After installing CA Spectrum 10.2.2 or above (ie, 10.2.3 or 10.3.x), model activation may take an extended amount of time.

Environment

Release: Spectrum 10.2.2 and above
Component: SPCCSS

Cause

SpectroSERVER model activation was taking 4 hours to activate due to new functionality introduced in 10.2.2: 

Condition Correlation of LACP ports
 
Starting from CA Spectrum 10.2.2 release, the LACP threshold violation alarms functionality is enhanced to avoid multiple independent alarms (on port and port channel) for bad links. With this enhancement, the alarms generated on the ports that are part of an LACP aggregation are correlated and suppressed. Only the LACP threshold violation alarms are generated on the port channels and the alarms generated on the ports are suppressed and shown as symptoms. 

If you have thousands of LACP ports, then you may have thousands of correlation domains.  To verify if this issue is affecting you:
1.  Navigate in a bash shell to the $SPECROOT/vnmsh directory
2.  Run:  ./connect
3.  Run:  ./show models | grep LACP | wc -l
4.  If you have thousands of LACP ports see how many correlation models you have:
5.  Run:  ./show models | grep Correlation | grep wc -l
6.  If the response shows over 25,000 entries, this issue may be affecting you.  When the SS starts it activates all models, and the correlation domain models take a long time to activate causing the activation delays.

Resolution

This functionality limits the amount of alarms that are generated when LACP port channels go down. If you do not need this functionality, you can disable it by adding the following entry to the $SPECROOT/SS/.vnmrc and stopping and restarting the SpectroSERVER:

lacp_corr_enable=false

If you do need this functionality, then a caveat to using it is to expect longer SS activation/startup times.