Upgrade Automation script hangs when validating Spectrum install

Products

• Network Observability
• Spectrum

Issue/Introduction

After the Spectrum upgrade has completed the Upgrade Automation scripts start the SpectroSERVER process and sends a heartbeat request to validate whether the SS is successfully running.  In some instances we have seen where the secondary SpectroSERVER may hang on startup due  to a db validation check with CORBA.  When this happens there is no response to the heartbeat so the script hangs.   

Cause

The secondary SpectroSERVER process is running but does not respond to the heartbeat request.

Resolution

On the secondary SpectroSERVER, kill the upgrade automation process which will allow the spectrum_operation_utils.sh to proceed on the orchestrator machine.

This will be fixed in a future release of the Upgrade Automation scripts.

Additional Information

If you experienced this, most likely you will need to stop/crash the SpectroSERVER process and reload the database.  This has only been seen on secondary SpectroSERVERs so you can remove the SS/.VNMDB.LOCK on the secondary SS and run an OnlineBackup on the primary SS.  Once complete start the secondary SS if the online backup processd did not start it.