UIM Spectrumgtw probe stops syncing alarms to Spectrum

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

We received the following error in Spectrum, and we haven't had any UIM alarms come through to Spectrum since:

CONTACT LOST WITH CA UIM's SPECTRUM GATEWAY PROBE

We increased logging for spectrumgtw probe using the following article:
How to enable trace level logging for the UIM Spectrum Gateway

Then we observed the following logs:

• In the alarm.log from spectrumgtw probe:
  • ERROR SpectrumGtwScheduler_Worker-9: [EmsServiceThread] [awaitingResponse] - Timeout happened while calling EMS Service, no response from EMS Service
• In the ems probe
  • we saw "Failure registering to maintenance_mode".
• In the maintenance_mode probe, we saw:
  • INFO  [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] cbRegister begin address /UIM_domainname/Primary_hub/Primary_robot/ems version null interval_start xxxxxxxxxx interval_stop xxxxxxxxxx
  • INFO  [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] cbRegister end address /UIM_domainname/Primary_hub/Primary_robot/ems version null interval_start xxxxxxxxxx interval_stop xxxxxxxxxx
  • INFO  [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] cbRegister begin address /UIM_domainname/Primary_hub/Primary_robot/nas version null interval_start xxxxxxxxxx interval_stop xxxxxxxxxx
  • INFO  [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] cbRegister end address /UIM_domainname/Primary_hub/Primary_robot/nas version null interval_start xxxxxxxxxx interval_stop xxxxxxxxxx
    
    along with the following for each of the above connection attempts:
  • WARN  [attach_socket, com.nimsoft.monitor.probe.MaintenanceModeProbe] Failure registering to maintenance_mode. 

Environment

Release : 20.3.2
Spectrumgtw 8.69HF2
Spectrum 22.2.6

Cause

Maintenance_mode probe failing to respond to EMS, preventing EMS from responding to Maintenance_mode.

Resolution

We increased the memory used by the maintenance_mode probe and restarted it.  Almost immediately, we started to see the alarms syncing to Spectrum.

From:

To:

The error messages in the maintenance_mode, ems, and spectrumgtw logs also stopped.