Online Backup Fault Tolerance Synchronization failure

Products

• CA Spectrum

Issue/Introduction

Online backups are not successful as on the Secondary SpectroSERVER, model activation is not finishing after 24 hours as seen in the VNM.OUT.

 

Mar 19 01:28:57 : SpectroSERVER has received shut down signal - scheduling shut down

Mar 19 01:28:57 : /opt/spectrum/SS/SpectroSERVER is shutting down...

Mar 19 01:28:59 : Closing all client connections...

Mar 19 01:28:59 : Bypassing CORBA shutdown...

Mar 19 01:28:59 : Stopping /opt/spectrum/SS/SpectroSERVER activity...

 

-----  NOTE  --------------------------------------------------------------

CA Technologies, A Broadcom Company recommends that the SpectroSERVER be

allowed to complete the shutdown process.

Database corruption may result if the SpectroSERVER is prematurely stopped.

-------------------------------------------------------------  NOTE  ------

 

Mar 19 01:28:59 :       waiting for model activates to complete...

Cause

By enabling Model Activation debug on the Secondary, we are able to see the model in conflict is a provider cloud model.

 

Mar 18 02:05:14  Model activation status - 613802/613805 activated( 99% ), 1 problematic.

currently processing 1 model activate triggers:

Model 0x206ba80 "ABC_ASN_1234"/Provider_Cloud - 18EVpnIHProviderCond - 1496 seconds

-----------------------------------------------------

Activate, 0x206ba80, (ABC_ASN_1234), 0x1048f, (Provider_Cloud), !First, 18EVpnIHProviderCond, 1   ,0.000000, 1516.607000, 0 ,

 

 

Environment

Release : 20.2 / 10.4.3.1

Component : Spectrum Core / SpectroSERVER  Online Backups

Resolution

Online Backup failed because Model Activation took more then 24 hours to complete (most likely stuck) and therefore the Secondary SS could not be brought down gracefully.

We enabled model activation debug and saw that a provider cloud model was a possible culprit.

 

There are already some enhancements included in 21.2.8 to resolve this

 

US753599 EVpnIHProviderCond Activation

F113511 Reduce SpectroSERVER Activation time & other Improvements

 

However as customer could not upgrade, we backported these enhancements into patch 10.04.03.01.PTF_10.4.3.103

If you are on 10.4.3.1 and cannot upgrade to 21.2.8, please open a support issue and request this patch.

Additional Information

To enable Model Activation Debug