How to enable Fault Isolation debugging in CA Spectrum.

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

Core switches were down and all the devices including core switches were suppressed. critical alerts was not generated on any devices. 

We have Unresolved Fault Alarm Disposition set to Device in Fault Domain from the VNM model, Fault Isolation Setttings.

Environment

DX NetOps 20.2 and later releases of Spectrum
Component : Spectrum Core / SpectroSERVER, Fault Isolation

Cause

Problem can be caused by not modeling every device and having an incomplete topology.  We can debug this further with the following debug.

Resolution

To enable debug first, get the FaultIsolation Model_Handle from CLI, as follows:

1. cd $SPECROOT/vnmsh/

2. Run:
   ./connect

3. Run:

    ./show models | grep FaultIsolation
   
   

4. You should see something similar to:
   
   0x100000b Fault Isolation 0x10413 FaultIsolation
   
   
5. To enable the debug when the problem is seen, run:
   
   

    ./update action=0x10084 mh=<fault isolation model handle> index=0,attr=0,type=3,val=2 
   
   

   For example:
   
   

   [spectrum@my_spectro:/opt/spectrum/vnmsh]$ ./update action=0x10084 mh=0x100000b index=0,attr=0,type=3,val=2

   update action: successful

   Response has 0 attributes:
   
   This should dump all of the fault domain data into the VNM.OUT where we can check further.
   
   

6. To disable the debug afterwards, run:
   
   

   ./update action=0x10084 mh=<fault isolation model handle> index=0,attr=0,type=3,val=0

    

Then provide the VNM.OUT with the debug, all of the events exported in CSV as far back as possible and a clear screenshot of the suppressed topology in a Support issue.