CORBA Request not accepted in Spectrum VNM

Products

• CA Spectrum
• CA eHealth

Issue/Introduction

Frequently we are receiving error "CORBA request not accepted after 60000ms" on control panel of MLS and it is hanging the whole spectrum java console.
Stopping and Starting spectroserver from control panel is resolving the issue temporarily but issue is again reappearing within 24-48 hours.

Environment

Release : all

Component : Spectrum Core / SpectroSERVER

Cause

Performance analysis showed red flag events were killing Spectrum performance.

Resolution

NOTE: Starting from DX NetOps Spectrum 21.2.4, the default root password for MySql is "MySqlR00t". For DX NetOps Spectrum versions prior to 21.2.4, the default root password is "root". In the following MySql commands, replace <PASSWD> with the root password for your DX NetOps Spectrum version.

The red flag events in question were device reconfiguration and trap storms.

Automatic reconfiguration should be turned off via OC on the worst offending devices.
Below we see some models with thousands of re-configurations over the 6 weeks of event data, which has a huge cist on the SS and OC.

Spectrum Event Summary for event 0x10050
#########################################################
Count: 16
Oldest Event Date: 3/15/20 17:24
Model Handle: 0x1016b8 Count: 4805
Model Handle: 0x1016b7 Count: 1607
Model Handle: 0x101887 Count: 10162
Model Handle: 0x1016bd Count: 92
Model Handle: 0x1016b5 Count: 12
Model Handle: 0x1016ae Count: 62
Model Handle: 0x1016aa Count: 112
Model Handle: 0x1016ad Count: 72

The VNM received 8,968 trap storms - that means they came from unmanaged devices. Turn off unmanaged trap handling on the VNM model, which alone should have a huge improvement on performance and enough to resolve the issue. However, those devices will continue to send traps. It may be 1 device or 1000 - we can query the ddmdb to see which devices caused the most.

- Log into the SpectroSERVER as the user that owns the Spectrum installation

- If on Windows, start a bash shell by running "bash -login"

- cd to the $SPECROOT/mysql/bin directory and enter the following command to log into mysql

Navigate to $SPECROOT/mysql/bin directory on the SpectroSERVER host

./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWD>  ddmdb -A

The following should be pasted into the mysql> prompt, each query starts with a SELECT and ends with a semi-colon (;):

# To get a count of the # of events that occured after a set date:
SELECT count(*) FROM event WHERE utime >= UNIX_TIMESTAMP("2008-01-01");

# To get the Top 10 events most commonly generated:
SELECT hex(type), COUNT(*) as cnt
FROM event GROUP BY type
ORDER BY cnt DESC LIMIT 10;

# To get the Top 10 models with the most events:
SELECT hex(e.model_h), m.model_name, COUNT(*) as cnt
FROM event e, model m WHERE e.model_h=m.model_h
GROUP BY e.model_h
ORDER BY cnt DESC LIMIT 10;

# To get the Top 10 high-volume days for events:
SELECT date(from_unixtime(utime)) as x, count(*) as cnt
FROM event GROUP BY x
ORDER BY cnt DESC LIMIT 10;

# To get the last 10 days volume of events:
SELECT date(from_unixtime(utime)) as x, count(*) as cnt
FROM event GROUP BY x
ORDER BY x DESC LIMIT 10;