Archive Manager does not start after upgrade to 21.2.8 onwards - SPC-OCP-10067
search cancel

Archive Manager does not start after upgrade to 21.2.8 onwards - SPC-OCP-10067

book

Article ID: 237081

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

Spectrum was upgraded to 21.2.8 on 03/08/2022.

The following error is displayed in the SCP (Spectrum Control Panel):


SPC-OCP-10067: The Archive Manager has been shutdown because there was an error opening its database. The most likely causes: 
 1. The SQL database server is not running. 
 2. The username, password, hostname, or database name are not correct. 
 3. The database is invalid or corrupt.


In the $SPECROOT/SS/DDM/ARCHMGR.OUT file:

Mar 16 21:45:45 : ArchMgr started as user 'xxxxx'
Mar 16 21:45:45 : ArchMgr validating database.
Error opening the DDM database. The most likely causes: 
 1. The SQL database server is not running. 
 2. The username, password, hostname, or database name are not correct. 
 3. localhost is not configured properly.
Mar 16 21:45:48 : ArchMgr error opening database.

Environment

Release : 21.2

Component : Spectrum Alarm

Cause

There were two problems:

a) The process was not set to the Spectrum Install Owner account

In the $SPECROOT/.installrc file:
OWNER=spectrum

b) Password mismatch in the $SPECROOT/SS/DDM/.configrc file

In the $SPECROOT/SS/DDM/.configrc file and removed the following line:

DDM_DATABASE_USERNAME=SPEC_admin
DDM_DATABASE_HOSTNAME=localhost
DDM_DATABASE_PASSWORD=spectrum

The Spectrum Install Owner account (spectrum in the above example) is no longer the valid password to connect to MySQL server. The SPEC_admin account has a different password.

Resolution

a) Followed the steps in this KB article to change the processd account:

https://knowledge.broadcom.com/external/article/38207/what-are-the-commands-for-reinstalling-t.html

Customer had to supply the password enclosed in double-quotes instead of single-quotes.


The ArchMgr is now started by the spectrum account, but the same errors are displayed in the ARCHMGR.OUT file:

Mar 16 hh:mm:ss : ArchMgr started as user 'spectrum'
Mar 16 hh:mm:ss : ArchMgr validating database.
Error opening the DDM database. The most likely causes: 
 1. The SQL database server is not running. 
 2. The username, password, hostname, or database name are not correct. 
 3. localhost is not configured properly.
Mar 16 hh:mm:ss : ArchMgr error opening database.

b) Edit the $SPECROOT/SS/DDM/.configrc file and removed the following line:

DDM_DATABASE_PASSWORD=spectrum


Successfully started the Archive Manager.

Additional Information

If the problem remains, please try logging as SPEC_admin in MySQL command prompt.

1. Open a bash shell (bash -login) logged as Spectrum Install owner account

2. Navigate to the $SPECROOT/mysq/bin/ directory

cd mysql/bin

3. Connect to the MySQL command prompt as SPEC_admin user:

./mysql -u<SpecAdminUser> -p<password> ddmdb

4. Run the following MySQL query:

select count(*) from event;

If you get something like this, reboot the Windows machine:

C:\Users\spectrum>bash -login
      0 [main] bash 5924 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0xEE0000)
bash: fork: retry: Resource temporarily unavailable
      1 [main] bash 8664 child_info_fork::abort: d:\win32app\Spectrum\NT-Tools\SRE\bin\cygiconv-2.dll: Loaded to different address: parent(0xF20000) != child(0x7A0000)
bash: fork: retry: Resource temporarily unavailable

 

In Spectrum 21.2.8 onwards, all the MySQL user passwords have been externalized in the $SPECROOT/mysql/data/myspectrumlogin.cnf file. These passwords are added to the myspectrumlogin.cnf file in an obfuscated format for security purposes. Furthermore, a new script (update_mysql_user.pl) is also available in the $SPECROOT/mysql/bin folder. You can use this script to update the password of the specific MySQL user.

In a fresh install of Spectrum 21.2.8 onwards, there is no longer the DDM_DATABASE_PASSWORD entry in the .configrc file. This will be fixed in the upcoming Spectrum 22.2.1

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/release-information/Features-and-Enhancements-2128.html

For more information, see: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/administrating/database-management/Update-MySQL-User-Passwords.html

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/administrating/oneclick-administration/oneclick-web-server-administration/configure-oneclick-mysql-server-passwords.html

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/programming/security-policy-statement/detailed-ca-spectrum-component-descriptions/mysql-password-storage.html

 

This has been fixed in DX NetOps Spectrum 22.2.1:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/22-2/release-information/issues-resolved.html

Symptom: In an upgraded DX NetOps Spectrum environment, DDM_DATABASE_PASSWORD is present in the $SPECROOT\SS\DDM\.configrc file, and Archive Manager is not starting.
Resolution: Code changes are done to ensure that DDM_DATABASE_PASSWORD no longer exists in the $SPECROOT\SS\DDM\.configrc file in an upgraded environment. This eventually facilitates the proper starting of Archive Manager. (DE539388, 33161158, 22.2.1)