PM Events not appearing in Spectrum - "queue is full(10000), attempt #1" message in Spectrum Tomcat logs

Products

• Network Observability
• Spectrum
• CA Performance Management

Issue/Introduction

Threshold events generated in DX NetOps CA Performance Management (CAPM) stop appearing in Spectrum after upgrade of both

Environment

DX NetOps Spectrum & CAPM 22.2.5 or later

Cause

In the Spectrum OneClick server log, we see hundreds of the following:

Rest subscription b5a8e7c3-f6e8-4cfa-9b2c-74a6c264b496 queue no longer full(9001), backlog is 1433
Rest subscription b5a8e7c3-f6e8-4cfa-9b2c-74a6c264b496 queue is full(10000), attempt #1

These are followed by hundreds of:

Sep 03, 2024 04:54:55.101 - IOException: Connection refused: connect

This can be caused by a hard timeout for the event subscription between Spectrum and PM. When this timeout is breached, the subscription is halted and then a new one is created. When this happens, the queue can become full and cause a slight delay in the event sync. 

Resolution

You can adjust the Event sync timeout as per the following:

TechDocs : DX NetOps 23.3 : Update the Configuration of the Alarm Service

Specifically, increase the RequestTimoutSec variable to stop this from happening as follows:

1. Using a REST client, target with a PUT request the Event Manager (EM) WebService
   
   
   http://<PC_HOST>:8281/EventManager/webservice/alarmservice/config
   
   
2. In the body of the PUT request, increase the value of RequestTimoutSec to 300 (default is 120 sec)
   
   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<AlarmServiceConfig>
  <Enabled>true</Enabled>
  <Parallelism>1</Parallelism>
  <RequestTimoutSec>300</RequestTimoutSec>
  <SubscriptionRetryDelaySec>300</SubscriptionRetryDelaySec>
  <SubscriptionBatchSize>1000</SubscriptionBatchSize>
  <SubscriptionHeartbeatIntervalSec>30</SubscriptionHeartbeatIntervalSec>
  <SubscriptionTimeoutMs>300000</SubscriptionTimeoutMs>
</AlarmServiceConfig>


3. Then you can run a GET request on the same URL to see if it has updated:
   
   http://<PC_HOST>:8281/EventManager/webservice/alarmservice/config

4. Restart the Spectrum OneClick server and the DX NetOps Portal services