SpectroSERVER crashed after DX NetOps 20.2 upgrade

book

Article ID: 195212

calendar_today

Updated On:

Products

CA Spectrum CA eHealth

Issue/Introduction

We have upgraded our Spectrum 10.2.1 to the latest NetOps 20.2. (Spectrum 10.4.2). Our DSS servers upgraded and are running fine. However, our MLS main location server will crash a few seconds after startup. The SS will not activate.

There is no CORE file and the $SPECROOT/SS/VNM.OUT file shows heap corruption:

*** Error in `/app/spectrum/SS/SpectroSERVER': corrupted size vs. prev_size: 0x000000000bc14c60 ***

Cause

What appears to have happened is a permissions issue on the server caused by the upgrade. NetOps 20.2 allows Spectrum to be installed as a non-root user. However older versions such as 10.2.1 this was not the case. Thus, in the original 10.2.1 install, many files (especially those for use in processd) were owned by root user. When the upgrade-in-place was done by non-root user, it created some permissions conflicts. This was the root cause of the crashing. 

Environment

Release : 10.4.2

Component : Spectrum Core / SpectroSERVER

Resolution

To solve the issue: 

1. run the provided script /Install-Tools/CASpectrum_ProcessRegister.sh as root user to reset installation permissions
    

2. Initialize database to $SPECROOT/SS/legacy.SSdb catalog

cd $SPECROOT/SS
run ../SS-Tools/SSdbload -ic legacy.SSdb


3. load the backup <db_filename>.SSdb models

../SS-Tools/SSdbload -m <db_filename>.SSdb


4. Switch the SpectroSERVER binary to the mts version

cd $SPECROOT/SS
mv SpectroSERVER SpectroSERVER.1042.orig
cp -p support/SpectroSERVER_mts ./SpectroSERVER


Note: verify that the owner/group and permissions match the SpectroSERVER.1042.orig we renamed earlier and modify if needed

This crash is resolved on 10.4.2.0.189 with patch 10.04.02.PTF_10.4.213.

This will be resolved in DX NetOps Spectrum 20.2.1 and above.

Additional Information

On a new installation of NetOps 20.2, or a fresh installation (migration upgrade) this is not an issue. 

If you are upgrading Spectrum from a version earlier than 10.4.0, recommend to run the install as root user: 

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/spectrum/10-4-2/installing-and-upgrading/fresh-install/installing-ca-spectrum.html