Excessive model re-configurations can cause CA Spectrum performance issues

Products

CA Spectrum DX NetOps

Issue/Introduction

For most device models, every poll cycle, Spectrum reads the following three attributes:

ifNumber

ifTableLastChange

ifStatusLastChange

If Spectrum detects the value of any one of the above attributes has changed since the last poll and the If_IsAutoCnfgActive attribute 0x11dd4 is enabled, it will automatically reconfigure the model to determine the change and update the model with the change.

This is very expensive in terms of CPU as Spectrum has to read a lot of snmp from the interfaces and ifMIBs.

This can also cause the memory to increase over time.

Spectrum will also log the 0x1001d event (below) which will specify the trigger that caused the model reconfiguration.

Event0001001d:

{d "%w- %d %m-, %Y - %T"} - Device {m} of type {t} has completed an interface reconfiguration. (event [{e}])

Interface Reconfiguration Trigger: {T InterfaceReconfigurationTriggers 1}

Interface Reconfiguration Status: {T CsInterfaceReconfigurationErrors 2}

The following are the possible triggers:

Initial-Reconfiguration - From initial model creation

Forced-Manual-Reconfiguration - Manual reconfiguration

Interface-Stack-Change-Reconfiguration - A change in ifStackLastChange

Interface-Table-Change-Reconfiguration - A change in ifTableLastChange

Interface-Count-Change-Reconfiguration - A change in ifNumber

Auto-Reconfiguration-Activated - When Auto Reconfiguration is enabled after being disabled

SNMP-Contact-To-Device-Re-established - After snmp contact is reestablished after snmp contact was lost

ICMP-SNMP-Contact-To-Device-Re-established - After icmp and snmp contact is reestablished after icmp and snmp contact was lost

Model-State-Changed - When a model state change has been detected

There is an Event Rate rule attached to the 0x1001d event which states if 6 0x1001d events are logged within 31 minutes, Spectrum will log event 0x10050 and assert a minor alarm on the device model.

Event00010050

{d "%w- %d %m-, %Y - %T"} - Device {m} of type {t} has triggered an interface reconfiguration every poll cycle for the past half hour. (event [{e}])

Environment

Spectrum - All releases.

Cause

In most cases, the root cause has been devices that dynamically create and remove interfaces that cause updates to the ifNumber, ifTableLastChange or ifStackLastChange attributes.

Resolution

Interface Reconfiguration can be handled Automatically within Spectrum Self-Health submenu.

Self-Health Monitoring

Excessive interface re-configurations handling:
Previously SS raised ‘Excessive Interface Reconfigure' alarm on a device if six interface re-configurations occur within 31 minutes. This frequent interface re-configurations adversely affect the SpectroSERVER's performance. To address this issue, from 10.3.2 onwards, if this option is enabled, SS disables auto-reconfiguration on such devices and waits for an age-out time (configurable to 1 hour by default). After the age-out time expires, SS enables the reconfiguration.

The interface reconfiguration functionality can be manually disabled by doing the following:

1. Setting If_IsAutoCnfgActive attribute 0x11dd4 to No. This will disable all interface reconfigurations for any reason. This can be set in any of the following ways:

a. In the Reconfiguration subview in the Information tab of the model by setting "Automatically Reconfigure Interfaces" to "No".

b. In the Attributes tab of the model by filtering for and selecting the If_IsAutoCnfgActive attribute 0x11dd4 to "No".

c. In the Attribute Editor by selecting "Automatically Reconfigure Interfaces" in the "Change Management" folder

d. Using the Command Line Interface (CLI) update command. Reference the "Command Line Interface User Guide" for information on using CLI.

2. If the Interface Reconfiguration Trigger is "Interface-Stack-Change-Reconfiguration", set the Use_If_Stack_Last_Change attribute id 0x130bc to "False" on the model using the Attributes tab, Attribute Editor or CLI.

3. If the Interface Reconfiguration Trigger is "Interface-Table-Change-Reconfiguration", set the Use_If_Table_Last_Change attribute id 0x11f7f to "False" on the model using the Attributes tab, Attribute Editor or CLI.