Active UIM Alarms Constantly Being Cleared and Recreated in Spectrum by the Spectrumgtw Probe

book

Article ID: 140537

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction

A UIM alarm is getting cleared on its own in Spectrum as per the events. Since the alarm is still alive in UIM the
  spectrumgtw probe is re-creating the alarm in Spectrum. After a short period of time, the alarm is again cleared
  in Spectrum and re-created. The clearing and re-creation cycle repeats continually.

 

 

Cause

It was found that the EventDisp (Event Disposition) files had been modified on the Spectrum side changing
  the severity so that it no longer matched UIM's Alarm Severity. The Specrtumgtw probe sees the mismatch
  in severity and attempts to rectify this by clearing and re-creating the alarm to try and keep them in sync. The
  modification on the Spectrum side prevents the alarm severities from ever matching so this process continues
  in an endless cycle while the alarm is still active in UIM


Spectrumgtw log Info

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - InitiateReconcilerAction START

Nov 12 2019 13:35:45,351 [pool-34-thread-1] DEBUG SpectrumGtwAlarmFilter - canBeFiltered start

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE      AlarmReconciler - Reconciler current alarm : AlarmAggregratorElem{SpectrumAlarm=Spectrum Alarm [  Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: null Alarm Trouble Ticket Id: null TargetProduct Alarm Id: FR64267104-80190 Alarm Manager : NAS Primary IPV4 Address: null Host Name  : uimrobota Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Alarm Severity: 1 Device Deleted Flag: false targetEntity : null Entity ID : 0xbc02f3 Modified Time : 0 Custom 1 : nullCustom 2 : nullCustom 3 : nullCustom 4 : nullCustom 5 : null UserTag 1 : Windows UserTag 2 : null Status(CauseCount):  RootCause , Symptoms : [] ], uimAlarm= Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.20 Alarm Trouble Ticket Id: TargetProduct Alarm Id: null Alarm Manager  : NAS Host Name  : primaryHubRobot Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :10.10.20.25 Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : 1.2.1:4 Status(CauseCount): RootCause, Symptoms : [] ], aggrEnum=UPDATE}

Nov 12 2019 13:35:45,351 [pool-34-thread-1] DEBUG SpectrumGtwAlarmFilter - canBeFiltered end

Nov 12 2019 13:35:45,351 [pool-34-thread-1] DEBUG SpectrumGtwAlarmFilter - shouldRemoveAlarm start

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateAlarms START

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateAlarms - update  SpectrumAlarm Spectrum Alarm [  Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: null Alarm Trouble Ticket Id: null TargetProduct Alarm Id: FR64267104-80190 Alarm Manager : NAS Primary IPV4 Address: null Host Name  : uimrobota Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Alarm Severity: 1 Device Deleted Flag: false targetEntity : null Entity ID : 0xbc02f3 Modified Time : 0 Custom 1 : nullCustom 2 : nullCustom 3 : nullCustom 4 : nullCustom 5 : null UserTag 1 : Windows UserTag 2 : null Status(CauseCount):  RootCause , Symptoms : [] ], uimAlarm= Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id: TargetProduct Alarm Id: null Alarm Manager  : NAS Host Name  : primaryHubRobot Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :10.10.20.25 Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : 1.2.1:4 Status(CauseCount): RootCause, Symptoms : [] ], aggrEnum=UPDATE}

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateSpectrumAlarm STARTSpectrum Alarm [  Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id:  TargetProduct Alarm Id: FR64267104-80190 Alarm Manager : NAS Primary IPV4 Address: null Host Name  : primaryHuba Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Alarm Severity: 4 Device Deleted Flag: false targetEntity : null Entity ID : D1992A49B544EB58D5F4BBBC983D410FE  Modified Time : 1573587264000 Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  Status(CauseCount):  RootCause , Symptoms : [] ]

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG  AlarmServiceFactory - getAlarmProducerService start

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG  AlarmServiceFactory - getAlarmProducerService returned com.nimsoft.probe.gateway.alarmservice.AlarmProducerProxy

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG   AlarmProducerProxy - Alarm  Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id: TargetProduct Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Alarm Manager  : NAS Host Name  : primaryHuba Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :null Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : null Status(CauseCount): RootCause, Symptoms : [] ]updated for proxy com.nimsoft.probe.gateway.alarmservice.AlarmProducerProxy

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateSpectrumAlarm END

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE      AlarmReconciler - updateAlarms - update targetProductAlarmId in UIM

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE      AlarmReconciler - updateAlarms - update alarm in spectrum  Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id: TargetProduct Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Alarm Manager  : NAS Host Name  : primaryHuba Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :10.10.10.18 Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : 1.2.1:4 Status(CauseCount): RootCause, Symptoms : [] ]

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateUIMAlarm START -  Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id: TargetProduct Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Alarm Manager  : NAS Host Name  : primaryHuba Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :10.10.10.18 Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : 1.2.1:4 Status(CauseCount): RootCause, Symptoms : [] ]

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - updateUIMAlarmExt Start Uim Alarm [  Alarm Id: FR64267104-80190 Troubleshooter: null Alarm Title: Service Control Manager(7038 - None): The wmiApSrv service was unable to log on as DOMAIN\User with the currently configured password due to the following error:  The request is not supported.  To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC). IP Address: 10.10.10.18 Alarm Trouble Ticket Id: TargetProduct Alarm Id: 5dcb0962-96c4-1176-008d-005056bd6b60 Alarm Manager  : NAS Host Name  : primaryHuba Source product  : NAS LandScape  : 11534336 Alarm Source: NAS Primary IPV4 Address :10.10.10.18 Alarm Severity: 4 Device Deleted Flag: false Information Warning Alarm deletion Flag: false Severity change for Informational or Warning Alarm: false Alarm severity change flagfalse Entity Id : D1992A49B544EB58D5F4BBBC983D410FE Custom 1 : Custom 2 : Custom 3 : Custom 4 : Custom 5 :  UserTag 1 : Windows UserTag 2 :  MetricTypeId : 1.2.1:4 Status(CauseCount): RootCause, Symptoms : [] ]

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE y.alarmservice.SeverityLevelMapping - Spectrum level of 4 is 2

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - resolveConflicts START  FR64267104-80190

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE y.alarmservice.SeverityLevelMapping - UIM level of 1 is 3

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - Alarm Severity is changed..Performing Reconciliation Action

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG ontroller.AlarmReconciliationAction - resolveSeverityConflicts start

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG ontroller.AlarmReconciliationAction - resolveSeverityConflicts end

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG      AlarmReconciler - resolveConflicts END

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG  AlarmServiceFactory - getAlarmProducerService start

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG  AlarmServiceFactory - getAlarmProducerService returned com.nimsoft.probe.gateway.alarmservice.SpectrumAlarmProducer

Nov 12 2019 13:35:45,351 [pool-34-thread-4] DEBUG SpectrumAlarmProducer - clear(IAlarm alarm) START

Nov 12 2019 13:35:45,351 [pool-34-thread-4] TRACE    SpectrumAlarmPush - Creating alarm batch for deletes. Start index 0. End Index 1

Nov 12 2019 13:35:45,414 [pool-34-thread-4] DEBUG    SpectrumAlarmPush - Sending Event Alarm Request 

Nov 12 2019 13:35:45,414 [pool-34-thread-4] TRACE           Connection - POST(ing): http://OneClickServer:80/spectrum/restful/eventalarm with payload object: 
 <?xml version="1.0" encoding="UTF-8"?><event-alarm-request xmlns="http://www.ca.com/spectrum/restful/schema/request">
<alarm-delete-list>
<alarm-delete uimid="FR64267104-80190"/>
</alarm-delete-list>
</event-alarm-request>

Environment

Spectrum 10.3.2
UIM 9.0.2
Spectrumgtw Probe 8.67HF2


Resolution

Modify the severity on the UIM Side to the desired level and remove the severity modification
   on the Spectrum side so that the alarm matches on both sides.