Archive Manager does not start after upgrade to 21.2.8 onwards - SPC-OCP-10067
Article ID: 237081
Updated On:
Products
Issue/Introduction
Spectrum was upgraded to 21.2.8 on 03/08/2022.
The following error is displayed in the SCP (Spectrum Control Panel):
SPC-OCP-10067: The Archive Manager has been shutdown because there was an error opening its database. The most likely causes:
1. The SQL database server is not running.
2. The username, password, hostname, or database name are not correct.
3. The database is invalid or corrupt.
In the $SPECROOT/SS/DDM/ARCHMGR.OUT file:
Mar 16 21:45:45 : ArchMgr started as user 'xxxxx'
Mar 16 21:45:45 : ArchMgr validating database.
Error opening the DDM database. The most likely causes:
1. The SQL database server is not running.
2. The username, password, hostname, or database name are not correct.
3. localhost is not configured properly.
Mar 16 21:45:48 : ArchMgr error opening database.
Environment
Release : 21.2
Component : Spectrum Alarm
Cause
There were two problems:
a) The process was not set to the Spectrum Install Owner account
In the $SPECROOT/.installrc file:
OWNER=spectrum
b) Password mismatch in the $SPECROOT/SS/DDM/.configrc file
In the $SPECROOT/SS/DDM/.configrc file and removed the following line:
DDM_DATABASE_USERNAME=SPEC_admin
DDM_DATABASE_HOSTNAME=localhost
DDM_DATABASE_PASSWORD=spectrum
The Spectrum Install Owner account (spectrum in the above example) is no longer the valid password to connect to MySQL server. The SPEC_admin account has a different password.
Resolution
a) Followed the steps in this KB article to change the processd account:
https://knowledge.broadcom.com/external/article/38207/what-are-the-commands-for-reinstalling-t.html
Customer had to supply the password enclosed in double-quotes instead of single-quotes.
The ArchMgr is now started by the spectrum account, but the same errors are displayed in the ARCHMGR.OUT file:
Mar 16 hh:mm:ss : ArchMgr started as user 'spectrum'
Mar 16 hh:mm:ss : ArchMgr validating database.
Error opening the DDM database. The most likely causes:
1. The SQL database server is not running.
2. The username, password, hostname, or database name are not correct.
3. localhost is not configured properly.
Mar 16 hh:mm:ss : ArchMgr error opening database.
b) Edit the $SPECROOT/SS/DDM/.configrc file and removed the following line:
DDM_DATABASE_PASSWORD=spectrum
Successfully started the Archive Manager.
Additional Information
If the problem remains, please try logging as SPEC_admin in MySQL command prompt.
1. Open a bash shell (bash -login) logged as Spectrum Install owner account
2. Navigate to the $SPECROOT/mysq/bin/ directory
$ cd mysql/bin
3. Connect to the MySQL command prompt as SPEC_admin user:
$ ./mysql -u<SpecAdminUser> -p<password> ddmdb
4. Run the following MySQL query:
select count(*) from event;
If you get something like this, reboot the Windows machine:
C:\Users\spectrum>bash -login
0 [main] bash 5924 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0xEE0000)
bash: fork: retry: Resource temporarily unavailable
1 [main] bash 8664 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0x7A0000)
bash: fork: retry: Resource temporarily unavailable
In Spectrum 21.2.8 onwards, all the MySQL user passwords have been externalized in the $SPECROOT/mysql/data/myspectrumlogin.cnf file. These passwords are added to the myspectrumlogin.cnf file in an obfuscated format for security purposes. Furthermore, a new script (update_mysql_user.pl) is also available in the $SPECROOT/mysql/bin folder. You can use this script to update the password of the specific MySQL user.
In a fresh install of Spectrum 21.2.8 onwards, there is no longer the DDM_DATABASE_PASSWORD entry in the .configrc file. This will be fixed in the upcoming Spectrum 22.2.1
For more information, see: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/administrating/database-management/Update-MySQL-User-Passwords.html
This has been fixed in DX NetOps Spectrum 22.2.1:
Symptom: In an upgraded DX NetOps Spectrum environment, DDM_DATABASE_PASSWORD is present in the $SPECROOT\SS\DDM\.configrc file, and Archive Manager is not starting.
Resolution: Code changes are done to ensure that DDM_DATABASE_PASSWORD no longer exists in the $SPECROOT\SS\DDM\.configrc file in an upgraded environment. This eventually facilitates the proper starting of Archive Manager. (DE539388, 33161158, 22.2.1)
Feedback
