ERRORS in ARCHMGR.OUT "Maximum vardata string length exceeded"

Products

• Spectrum
• Network Observability

Issue/Introduction

We are seeing these errors in the ARCHMGR.OUT log file

SPC-SHD-29021:
**************************************************************************
The following output is intended to facilitate debugging, please forward
to CA technical support.  The files listed are not part of the customer
installation of SPECTRUM.
**************************************************************************

Apr 16 19:07:14 ERROR TRACE at ModelArchDBImp.cc(1572): ModelArchDBImp::submitEvent ( model_h=0x3ea56d04, type=0x82105a, utime: 1587056834 ): Maximum vardata string length exceeded
Apr 16 20:20:18 ERROR TRACE at ModelArchDBImp.cc(1572): ModelArchDBImp::submitEvent ( model_h=0x3eac7752, type=0x82105a, utime: 1587061218 ): Maximum vardata string length exceeded
Apr 16 20:35:21 ERROR TRACE at ModelArchDBImp.cc(1572): ModelArchDBImp::submitEvent ( model_h=0x3ea56e15, type=0x82105a, utime: 1587062121 ): Maximum vardata string length exceeded
Apr 16 21:35:02 ERROR TRACE at ModelArchDBImp.cc(1572): ModelArchDBImp::submitEvent ( model_h=0x3eac7706, type=0x82105a, utime: 1587065702 ): Maximum vardata string length exceeded

Environment

DX NetOps Spectrum Release : Any
Spectrum Archive Manager

 

Cause

The default database storage for vardata_string for text (65k) was not large enough to accommodate large changes in device configuration (event type 0x82105a).

Changes were made back in Spectrum 10.2 to increase the default database storage for default vardata_string from text (65k) to mediumtext (16M). If the ddmdb database was not initialized, then this field has not changed and must be done manually,

This can be verified by performing the following query on the ddmdb database from mysql and checking that the vardata_string is set to mediumtext.

mysql> desc event;
+----------------+----------------------+------+-----+---------+-------+
| Field          | Type                 | Null | Key | Default | Extra |
+----------------+----------------------+------+-----+---------+-------+
| model_h        | int(10) unsigned     | NO   | PRI | NULL    |       |
| utime          | int(11)              | NO   | PRI | NULL    |       |
| counter        | int(10) unsigned     | NO   | PRI | NULL    |       |
| clk_seq        | smallint(5) unsigned | NO   | PRI | NULL    |       |
| version        | tinyint(4)           | NO   | PRI | NULL    |       |
| node_id        | char(6)              | NO   | PRI | NULL    |       |
| user_key       | int(10) unsigned     | NO   |     | NULL    |       |
| type           | int(10) unsigned     | NO   | MUL | NULL    |       |
| severity       | int(10) unsigned     | NO   |     | NULL    |       |
| vardata_string | text                 | YES  |     | NULL    |       | <<-- Needs to be set to "mediumtext", NOT "text"
+----------------+----------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

Resolution

NOTE: Starting from DX NetOps Spectrum 21.2.4, the default root password for MySql is "MySqlR00t". For DX NetOps Spectrum versions prior to 21.2.4, the default root password is "root". In the following MySql commands, replace <PASSWORD> with the root password for your DX NetOps Spectrum version.

If the DDMDB was not reinitialized then use the following steps to manually make this change:

1. Log into the SpectroSERVER systems as the user that owns the Spectrum installation
   
   
2. If on Windows, start a bash shell by running "bash -login"
   
   
3. Find the process id of the ArchMgr process
   
   
4. cd to the $SPECROOT/bin directory and run the following command to stop the Archive Manager where <PID> is the ArchMgr Process ID noted previously:
   
   ./cmdC localhost 2 <PID>


5. Verify the ArchMgr process is not running
   
   
6. cd to the $SPECROOT/mysql/bin directory and enter the following command to log into mysql:
   
   ./mysql --defaults-file=../my-spectrum.cnf -uroot -p<PASSWORD> ddmdb
   
   
7. Enter the following command at the mysql prompt:
   
   ALTER table event MODIFY vardata_string mediumtext;
   
   
8. Enter the following command to verify the vardata_string is set to mediumtext

desc event;
   
   
9. cd to the $SPECROOT/bin and enter the following command to start the Archive Manager: (This command will see there is already a SpectroSERVER running and then start the Archive Manager.)
   
   ./startSS.pl


10. Monitor the ARCHMGR.OUT file to see if the message reappear.

Additional Information

If performing the steps manually please plan appropriate ArchMgr downtime.  In our test environment, a 6GB database took over an hour to modify.