CONTACT LOST TO SECONDARY SPECTROSERVER 0x10c0e alarm is not cleared consistently

Products

• Network Observability
• Spectrum

Issue/Introduction

In the Component Detail Events tab for the SpectroSERVER claims that the alarm is cleared after the OLB (Online Backup) but in the Contents Alarms tab the alarm is still show. This is a random issue, sometimes the alarm is cleared successfully.

The alarm is still active on the SpectroSERVER.

$ ./show alarms | grep 0x10c0e

130628706  09/05/2025 00:02:56  0x10c0e    0x1500007  <SpectroSERVER hostname>                  LocalLscpe      MAJOR        09/05/2025 00:02:56  No  No

 

Turn on event tracing.

On the SpectroSERVER host.

cd $SPECROOT/vnmsh

./connect

./update action=0x10274 mh=<VNM model> index=0,attr=0,type=15,val=0x10c21

./update action=0x10274 mh=<VNM model> index=0,attr=0,type=15,val=0x10c0e

Reproduce the issue by running the Online Database Backup.

Once the issue is reproduced, disable the event tracing.

./update action=0x10275 mh=<VNM model> index=0,attr=0,type=15,val=0x10c21

./update action=0x10275 mh=<VNM model> index=0,attr=0,type=15,val=0x10c0e

 

The issue did not occur on-demand OLB:

Oct 01 12:01:23 EMS TRACE at CsEvDispTbl.cc(2777): EMS is processing an event:
Oct 01 12:01:23 EMS TRACE at CsEvDispTbl.cc(2781):   event code: 0x10c0e
Oct 01 12:01:23 EMS TRACE at CsEvDispTbl.cc(2782):   model handle: 0x1500007
Oct 01 12:01:23 EMS TRACE at CsEventDispActionCreateAlarm.cc(203):     Processing create alarm action
Oct 01 12:01:23 EMS TRACE at CsEventDispActionLog.cc(109):     Processing log action
Oct 01 12:04:31 EMS TRACE at CsEvDispTbl.cc(2777): EMS is processing an event:
Oct 01 12:04:32 EMS TRACE at CsEvDispTbl.cc(2781):   event code: 0x10c21
Oct 01 12:04:32 EMS TRACE at CsEvDispTbl.cc(2782):   model handle: 0x1500007
Oct 01 12:04:32 EMS TRACE at CsEventDispActionClearAlarm.cc(79):     Processing clear action
Oct 01 12:04:32 EMS TRACE at CsEventDispActionLog.cc(109):     Processing log action
Oct 01 12:04:32 EMS TRACE at CsEvDispTbl.cc(2777): EMS is processing an event:
Oct 01 12:04:32 EMS TRACE at CsEvDispTbl.cc(2781):   event code: 0x10c21
Oct 01 12:04:32 EMS TRACE at CsEvDispTbl.cc(2782):   model handle: 0x1500007
Oct 01 12:04:32 EMS TRACE at CsEventDispActionClearAlarm.cc(79):     Processing clear action
Oct 01 12:04:32 EMS TRACE at CsEventDispActionLog.cc(109):     Processing log action

 

The issue was reproduced on the scheduled OLB.

Oct 02 00:03:09 EMS TRACE at CsEvDispTbl.cc(2777): EMS is processing an event:
Oct 02 00:03:09 EMS TRACE at CsEvDispTbl.cc(2781):   event code: 0x10c0e
Oct 02 00:03:09 EMS TRACE at CsEvDispTbl.cc(2782):   model handle: 0x1500007
Oct 02 00:03:09 EMS TRACE at CsEventDispActionCreateAlarm.cc(203):     Processing create alarm action
Oct 02 00:03:09 EMS TRACE at CsEventDispActionLog.cc(109):     Processing log action

The 0x10c21 event is missing to clear the alarm.

Environment

DX NetOps Spectrum 24.3.5 in RHEL 9

Resolution

Added the following entries below on the SSs. If the entries already exist, update them to match the values below. If they do not exist, add them.

sync_all_ext_interface_attrs=false
int_copyattr_enable=true
failed_cache_for_unmanaged_trap=true
tds_local_v3_profiles_store=true
v3_profile_hash_factor=16384
fix_v3_profile_lock=false
read_vrf_interfaces=false

Stop/start the SS to put the change in place.

 

Reference Articles:
https://knowledge.broadcom.com/external/article?articleId=411244
https://knowledge.broadcom.com/external/article?articleId=275841

Additional Information

Contact lost to secondary alarms happen consistently - 0x10c0e event
https://knowledge.broadcom.com/external/article?articleNumber=131917