Excessive events can cause performance issues in Spectrum

book

Article ID: 195716

calendar_today

Updated On:

Products

CA Spectrum CA eHealth

Issue/Introduction

Excessive events can cause performance issues in Spectrum.

Spectrum can get so busy with trying to process events that is stops responding to client requests like AlarmNotifier or OneClick causing client to switch to the econdary SpectroSERVER in a Fault Tolerant environment.

Environment

Release : Any

Component : Spectrum Core / SpectroSERVER

Resolution

You can query the ddmdb database on the SpectroSERVER system to identify the what events have been logged over a period of time, how many of the individual events were logged, what models logged the events and how many events the individual model logged.

Then, based on the output, investigate why so many events are being logged.

1. Log into the SpectroSERVER system as the user that owns the Spectrum installation

2. If on Windows, start a bash shell by running "bash -login"

3. cd to the $SPECROOT/mysql/bin directory

4. Run the following mysql commands:

    a. To find out what events have been logged and how many, run the following mysql command changing the start/end date/time as needed:

./mysql --defaults-file=../my-spectrum.cnf -uroot -proot ddmdb -e "SELECT hex (type), count(*) as count from event where utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') group by type order by count desc;" 

    b. From the output of the above command, the first column of data will specify the hex value of the event id. Replace <EVENT_ID> in the below command with this value and set the start/end date/time to the same value to find out what models logged that event during the same time period and how many times that model logged the event:

./mysql --defaults-file=../my-spectrum.cnf -uroot -proot ddmdb -e " SELECT hex(model_h), count(*) as count from ddmdb.event where utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') and hex (type)=<EVENT_ID> group by hex(model_h) order by count desc;" 

    c. Run the following mysql command to get an overall list of events logged by what models and how many were logged by the model over the time period specified:

./mysql --defaults-file=../my-spectrum.cnf -uroot -proot ddmdb -e "select hex( type ), hex( e.model_h ), m.model_name, count( * ) as cnt from event e, model m where e.model_h = m.model_h and utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') group by type, e.model_h order by cnt desc;"

Additional Information

The following is an example:

After running the mysql command from step 4a above, we see the following output:

hex (type) count
10701 397701
10801 392115
10702 226007
210017 161847
220001 155133
220002 155093
220004 138763

The first column of data is the hex value of the event id.

The second column of data is the number of times that event was generated in the time frame specified.

In the above example output, the 10701 (0x10701) event was generated 397,701 times in the time frame specified.

On the SpectroSERVER system, you can find out what the 0x10701 event means by examining the associated Event Format file located in the $SPECROOT/SG-Support/CsEvFormat directory or, in the case of a custom event id, the $SPECROOT/custom/Events/CsEvFormat directory.

For the 0x10701 event, the Event Format file is the Event00010701_en_US file located in the $SPECROOT/SG-Support/CsEvFormat directory. The following is the contents of this file:

Event00010701_en_US
{d "%w- %d %m-, %Y - %T"} - Alarm number {I 0x0700} with probable cause id {H 3} generated for device {m} of type {t}. The severity of this alarm is {T Condition 2}. (event [{e}])

The next step is to find out what models generated the events from the previous command.

After running the mysql command from step 4b, using the event id in this example, the 10701 event id, we see the following:

hex(model_h) count
2BB667 73231
2BB7E1 60762
2C43A1 53509
2BBAD5 34720
2BB5A8 32531
2C46B7 16614
2C5E08 15438
2BB89E 10274

The first column of data is the model handle in the Spectrum database that generated the event.

The second column of data is the number of time that model generated the event in the time frame specified.

In the above example output, the model at model handle 2BB667 (0x2bb667) generated the 10701 event 73,231 time in the time frame specified.

To find the exact model, you can use the Command Line Interface (CLI) on the SpectroSERVER system where this device is modeled.

1. Log into the SpectroSERVER system as the user that owns the Spectrum installation

2. If on Windows, start a bash shell by running "bash -login"

3. cd to the $SPECROOT/vnmsh directory and enter the following command to start a CLI session:

./connect

4. Enter the following command where <MH> is the model handle from the mysql output:

./show models | grep -i <MH>

Using the above model handle as an example, the command would be:

./show models | grep -i 2BB667 

The following is an example of the output:

./show models | grep -i 0x2bb667
0x2bb667   Sim15090:Lab-A1                   0x21000c    Rtr_Cisco

Now you can examine the device for this model to determine why the model is generating these events.

Recommendation: Place the model into Maintenance Mode to prevent further events from being generated until the issue can be addressed as to why.