Spectrum Report Manager (SRM) Last Event time is not updating for a landscape
search cancel

Spectrum Report Manager (SRM) Last Event time is not updating for a landscape

book

Article ID: 262370

calendar_today

Updated On:

Products

Network Observability Spectrum

Issue/Introduction

In SRM, SpectroSERVER is not processing any more Events

Environment

DX NetOps Spectrum Release : 20.2 or later

Cause

SRM is fetching 10007 events from the landscape and encountering CORBA DOES_NOT_EXIST exceptions from ArchiveManager.

When this occurs, SRM retries the event fetch with the last successfully fetched event time as the start-time, but in this case the last successful event time is  2023-02-07 20:36:35.0.

When looking at the output shown in the OneClick server log, it looks like the 10007 events it fetched during this cycle all were raised at the same time - 2023-02-07 20:36:35.0 . Hence its not able to proceed further.

Line    861: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Initializing instant event poller (Domain: spectroSERVER01)
    Line    862: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Initializing event poller (Domain: spectroSERVER01)
    Line    863: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - SRM to begin processing events for domain spectroSERVER01.
    Line    890: Feb 17, 2023 09:22:25.341 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line    892: Feb 17, 2023 09:22:25.357 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line    900: Feb 17, 2023 09:22:26.728 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  41354: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  41356: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  41358: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  41361: Feb 17, 2023 12:23:36.377 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  41362: Feb 17, 2023 12:23:37.848 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  47326: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  47328: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  47330: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  47333: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  47334: Feb 17, 2023 15:14:12.955 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  53657: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  53659: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  53661: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  53664: Feb 17, 2023 18:08:55.697 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  53665: Feb 17, 2023 18:08:56.667 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  60073: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  60075: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  60077: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  60080: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  60081: Feb 17, 2023 20:59:03.798 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  66546: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  66548: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  66550: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  66553: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  66554: Feb 17, 2023 23:50:21.496 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007

Resolution

To overcome this situation, SRM needs to fetch events that were generated after 2023-02-07 20:36:35.0 in the above example, hence we need to fetch more events in the single.

OOB ArchiveManager returns only 10007 events for every request from any client (OneClick client, SRM, CAPM Integ code etc…). SRM polls for events every 5 minutes (5 mins interval for Instant Polling & 1 hr Interval for all other events polling including Outage events) and takes more time to synchronize hence we can increase the record_search_limit on the Archive Manager side to increase this events limit.

So change the threshold limit from 10007 to 50000 .

Do the following on the affected landscape to increase the record search limit: 

  1. Stop the Archive Manager. 

  2. Edit $SPECROOT\SS\DDM\.configrc 

  3. Add record_search_limit=50000 and save it. 

  4. Restart the Archive Manager.

 

Now with this configuration, it will return 50000 events for every requested client.

Note: Once sync gets completed you can revert to 10007 or remove the above added entry to avoid any performance-related issues.

Powered by Wolken Software