"Rest subscription queue is full" errors seen in the OneClick log file

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

The following error is seen in the OneClick $SPECROOT/logs/stdout.log (Windows) or catalina.out (Linux) file:

Rest subscription cce46672-dc08-4e75-8633-020ffc11d142 queue is full(10000), attempt #1
Rest subscription 7213b1d9-bf0f-409f-9fbe-64725887b4bf queue is full(10000), attempt #2
Rest subscription 22948e8e-9fd2-4a60-83b7-9f02d08a63b8 queue is full(10000), attempt #1
Rest subscription 1bda843f-d126-44e8-a82c-089d35a4fcd8 queue is full(10000), attempt #1

This is causing OneClick performance issues including high memory.

Environment

Release : Any

Component : Spectrum OneClick/Web API

Cause

The REST alarm subscription for UIM calls all alarms and all alarm attributes.  When there are thousands of alarms there are thousands of entries for REST to process.  The default alarm synchronization time is too short, not allowing the queue to be fully processed.  This leads to cpu and memory issues with the OneClick Tomcat java process.

 

Resolution

Increase the Full Alarm Synchronization time in the Spectrum Gateway Probe configuration.  The recommendation is to set it to 12 hrs.  The incremental sync will take care of the updates.  12 hours will be more than enough time to allow tomcat to process the REST alarms.

The REST Subscription queue is full error will still show in the logs at the time of alarm synchronization.  This is just an informational message and can be ignored.

Additional Information

Please reference the "subscription" section of the Spectrum documentation for more information.