Archive Manager errors or VNM performance issues seen after upgrading to Spectrum 24.3.7 or higher

Products

• Network Observability
• Spectrum

Issue/Introduction

You have noticed either or both of the following issues  after upgrading to Spectrum 24.3.7 or higher

• intermittent issues with Archive Manager and during times, for no apparent reason.  You see the following errors and Archive Manager becomes unusable.

 

Jul 02 01:44:00 : Closing database
mysql_real_connect: Connection failure -  Error: Access denied for user '########'@'localhost' (using password: NO)
Jul 02 01:44:00 ERROR TRACE at DDMDbIf.cc(2377): Error while terminating open statistics series.
ModelArchDb ERROR at DDMDbIf.cc(2378):
Generic Error: 1, checkAndReconnect/checkAndReconnect: Connection failure
Native Error: 0,
mysql_real_connect: Connection failure -  Error: Access denied for user '########'@'localhost' (using password: NO)
Jul 02 01:44:00 ERROR TRACE at DDMDbIf.cc(2387): Error while updating overdue_chkpt_utime to NOW.
ModelArchDb ERROR at DDMDbIf.cc(2388):
Generic Error: 1, checkAndReconnect/checkAndReconnect: Connection failure
Native Error: 0,
mysql_real_connect: Connection failure -  Error: Access denied for user '########'@'localhost' (using password: NO)
Jul 02 01:44:00 ERROR TRACE at CsLogMgr.cc(700): database.close error

*********************************************

Symptom seen when this is happening:

Since ARCHMGR is effectively hung, no new event is created when a model is added to the DefaultDomain GC, 

Since that event is what triggers the new device to get synched to Portal, Devices are not getting synched to portal.

NOTE: 

The events icon in OneClick shows as green, even though it is not working.

 

• You see requests with long processing times and requests not being processed in VNM.OUT leading to CORBA rejects, server unavailability including landscape service switchover to secondary in OC. Secondary SpectroSERVER is affected too.

Oct 06 15:30:08 WARNING at ItcQdItem.cc(374): Request 2557540 accepted, but not complete after 60000ms, cs delta=57204. Triggering moot trace.
Oct 06 15:30:08 WARNING at ItcQdItem.cc(374): Request 2557543 accepted, but not complete after 60000ms, cs delta=57185. Triggering moot trace.
Oct 06 15:40:09 WARNING at ItcQdItem.cc(374): Request 2678544 accepted, but not complete after 60000ms, cs delta=49720. Triggering moot trace.
Oct 06 15:40:12 WARNING at ItcQdItem.cc(374): Request 2679265 accepted, but not complete after 60000ms, cs delta=60371. Triggering moot trace.

 

Environment

Spectrum 24.3.8+

 

Cause

Code change needed to be done on the Spectrum side to fix for MYSQL 8.

Resolution

This has been fixed in the 25.4.1 release of Spectrum.

 

We have patches available for these specific Spectrum releases.

 

* 24.3.09_D310

* 24.3.10_D311

* 24.3.11_D313

* 24.3.12.D315

* 24.3.13_D312

 

 

If you have one of these releases please open a support case along with your $SPECROOT/Install-Tools/.history file and request this patch.

Additional Information

Restarting the archive manager will fix the problem temporarily but it will eventually be seen again.

su - <SPECOWNER>

stop archmgr process 

cd <$SPECROOT>/bin

./cmdC localhost 2 ARCHMGR ARCHMGR.OUT

 

cd ../SS/DDM

Remove the .DDMDB.LOCK file if present and Archive Manager is not running

ls -al will show if .DDMDB.LOCK is there, 

rm -f .DDMDB.LOCK

 

start the archive manager

cd <SPECROOT>/bin

./launchinstdbapp localhost ARCHMGR y ARCHMGR.OUT