Excessive events can cause performance issues in Spectrum
Article ID: 195716
Updated On:
Products
Issue/Introduction
Excessive events can cause performance issues in Spectrum resulting in high CPU and memory issues.
Spectrum can get so busy with trying to process events that is stops responding to client requests like AlarmNotifier or OneClick causing client to switch to the secondary SpectroSERVER in a Fault Tolerant environment.
Environment
Release : Any
Component : Spectrum Core / SpectroSERVER
Resolution
You can query the ddmdb database on the SpectroSERVER system to identify the what events have been logged over a period of time, how many of the individual events were logged, what models logged the events and how many events the individual model logged.
Then, based on the output, investigate why so many events are being logged.
1. Log into the SpectroSERVER system as the user that owns the Spectrum installation
2. If on Windows, start a bash shell by running "bash -login"
3. cd to the $SPECROOT/mysql/bin directory
4. Run the following mysql commands:
a. To find out what events have been logged and how many, run the following mysql command changing the start/end date/time as needed:
./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> ddmdb -e "SELECT hex (type), count(*) as count from event where utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') group by type order by count desc;"
b. From the output of the above command, the first column of data will specify the hex value of the event id. Replace <EVENT_ID> in the below command with this value and set the start/end date/time to the same value to find out what models logged that event during the same time period and how many times that model logged the event:
./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> ddmdb -e " SELECT hex(model_h), count(*) as count from ddmdb.event where utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') and hex (type)='<EVENT_ID>' group by hex(model_h) order by count desc;"
c. Run the following mysql command to get an overall list of events logged by what models and how many were logged by the model over the time period specified:
./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD> ddmdb -e "select hex( type ), hex( e.model_h ), m.model_name, count( * ) as cnt from event e, model m where e.model_h = m.model_h and utime > UNIX_TIMESTAMP('2020-07-20 00:00:00') and utime < UNIX_TIMESTAMP('2020-07-21 00:00:00') group by type, e.model_h order by cnt desc;"
Additional Information
The following is an example:
After running the mysql command from step 4a above, we see the following output:
hex (type) count
10701 397701
10801 392115
10702 226007
210017 161847
220001 155133
220002 155093
220004 138763
The first column of data is the hex value of the event id.
The second column of data is the number of times that event was generated in the time frame specified.
In the above example output, the 10701 (0x10701) event was generated 397,701 times in the time frame specified.
On the SpectroSERVER system, you can find out what the 0x10701 event means by examining the associated Event Format file located in the $SPECROOT/SG-Support/CsEvFormat directory or, in the case of a custom event id, the $SPECROOT/custom/Events/CsEvFormat directory.
For the 0x10701 event, the Event Format file is the Event00010701_en_US file located in the $SPECROOT/SG-Support/CsEvFormat directory. The following is the contents of this file:
Event00010701_en_US
{d "%w- %d %m-, %Y - %T"} - Alarm number {I 0x0700} with probable cause id {H 3} generated for device {m} of type {t}. The severity of this alarm is {T Condition 2}. (event [{e}])
The next step is to find out what models generated the events from the previous command.
After running the mysql command from step 4b, using the event id in this example, the 10701 event id, we see the following:
hex(model_h) count
2BB667 73231
2BB7E1 60762
2C43A1 53509
2BBAD5 34720
2BB5A8 32531
2C46B7 16614
2C5E08 15438
2BB89E 10274
The first column of data is the model handle in the Spectrum database that generated the event.
The second column of data is the number of time that model generated the event in the time frame specified.
In the above example output, the model at model handle 2BB667 (0x2bb667) generated the 10701 event 73,231 time in the time frame specified.
To find the exact model, you can use the Command Line Interface (CLI) on the SpectroSERVER system where this device is modeled.
1. Log into the SpectroSERVER system as the user that owns the Spectrum installation
2. If on Windows, start a bash shell by running "bash -login"
3. cd to the $SPECROOT/vnmsh directory and enter the following command to start a CLI session:
./connect
4. Enter the following command where <MH> is the model handle from the mysql output:
./show models | grep -i <MH>
Using the above model handle as an example, the command would be:
./show models | grep -i 2BB667
The following is an example of the output:
./show models | grep -i 0x2bb667
0x2bb667 Sim15090:Lab-A1 0x21000c Rtr_Cisco
Now you can examine the device for this model to determine why the model is generating these events.
Recommendation: Place the model into Maintenance Mode to prevent further events from being generated until the issue can be addressed as to why.
Feedback
