Archive Manager does not start after upgrade to 22.2.x or later - "SPC-OCP-10067"

Products

• Spectrum
• Network Observability

Issue/Introduction

After Spectrum upgrade to 22.2.x or later, the following error is displayed in the SCP (Spectrum Control Panel):

SPC-OCP-10067: The Archive Manager has been shutdown because there was an error opening its database. The most likely causes: 
 1. The SQL database server is not running. 
 2. The username, password, hostname, or database name are not correct. 
 3. The database is invalid or corrupt.

In the $SPECROOT/SS/DDM/ARCHMGR.OUT file, the following is observed:

Mar 16 21:45:45 : ArchMgr started as user 'xxxxx'
Mar 16 21:45:45 : ArchMgr validating database.
Error opening the DDM database. The most likely causes: 
 1. The SQL database server is not running. 
 2. The username, password, hostname, or database name are not correct. 
 3. localhost is not configured properly.
Mar 16 21:45:48 : ArchMgr error opening database.

Environment

DX NetOps Spectrum Release : 22.2.x/23.3.x 

Cause

This is caused by two potential issues:

1. The process was not set to the Spectrum Install Owner account. In the $SPECROOT/.installrc file:
   
   OWNER=spectrum
   
   
2. Password mismatch in the $SPECROOT/SS/DDM/.configrc file, shown in the highlighted line below:
   
   DDM_DATABASE_USERNAME=SPEC_admin
DDM_DATABASE_HOSTNAME=localhost
DDM_DATABASE_PASSWORD=spectrum

The Spectrum Install Owner account password, DDM_DATABASE_PASSWORD (spectrum in the above example) is no longer valid to connect to MySQL server.

Resolution

1. Follow the steps in this KB article to change the processd account:
   
   KB : What are the commands for reinstalling the CA Spectrum processd service due to a user password change on Windows?
   
   The ArchMgr is now started by the spectrum account, but the same errors are displayed in the ARCHMGR.OUT file:
   
   Mar 16 hh:mm:ss : ArchMgr started as user 'spectrum'
Mar 16 hh:mm:ss : ArchMgr validating database.
Error opening the DDM database. The most likely causes:
 1. The SQL database server is not running.
 2. The username, password, hostname, or database name are not correct.
 3. localhost is not configured properly.
Mar 16 hh:mm:ss : ArchMgr error opening database.
   
   
2. Edit the $SPECROOT/SS/DDM/.configrc file and remove the following line:
   
   DDM_DATABASE_PASSWORD=spectrum

This should allow the Archive Manager to successfully start.

Additional Information

If the problem remains, please try logging as SPEC_admin in MySQL command prompt.

1. Open a bash shell (bash -login) logged as Spectrum Install owner account
   
   
2. Navigate to the $SPECROOT/mysq/bin/ directory
   
   $ cd mysql/bin
   
   
3. Connect to the MySQL command prompt as SPEC_admin user:
   
   $ ./mysql -u<SpecAdminUser> -p<password> ddmdb


4. Run the following MySQL query:
   
   select count(*) from event;

If you get something like this, reboot the Windows machine:

C:\Users\spectrum>bash -login
      0 [main] bash 5924 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0xEE0000)
bash: fork: retry: Resource temporarily unavailable
      1 [main] bash 8664 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0x7A0000)
bash: fork: retry: Resource temporarily unavailable

 

In Spectrum 21.2.8 onwards, all the MySQL user passwords have been externalized in the $SPECROOT/mysql/data/myspectrumlogin.cnf file. These passwords are added to the myspectrumlogin.cnf file in an obfuscated format for security purposes. Furthermore, a new script (update_mysql_user.pl) is also available in the $SPECROOT/mysql/bin folder. You can use this script to update the password of the specific MySQL user.

In a fresh install of Spectrum 21.2.8 onwards, there is no longer the DDM_DATABASE_PASSWORD entry in the .configrc file.