Spectrum Report Manager (SRM) Last Event time is not updating for a landscape
search cancel

Spectrum Report Manager (SRM) Last Event time is not updating for a landscape

book

Article ID: 262370

calendar_today

Updated On:

Products

DX NetOps CA Spectrum

Issue/Introduction

In SRM, SpectroSERVER is not processing any more Events

Environment

DX NetOps Spectrum Release : 20.2 or later

Cause

SRM is fetching 10007 events from the landscape and encountering CORBA DOES_NOT_EXIST exceptions from ArchiveManager.

When this occurs, SRM retries the event fetch with the last successfully fetched event time as the start-time, but in this case the last successful event time is  2023-02-07 20:36:35.0.

When looking at the output shown in the OneClick server log, it looks like the 10007 events it fetched during this cycle all were raised at the same time - 2023-02-07 20:36:35.0 . Hence its not able to proceed further.

Line    861: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Initializing instant event poller (Domain: spectroSERVER01)
    Line    862: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Initializing event poller (Domain: spectroSERVER01)
    Line    863: Feb 17, 2023 09:22:21.808 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - SRM to begin processing events for domain spectroSERVER01.
    Line    890: Feb 17, 2023 09:22:25.341 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line    892: Feb 17, 2023 09:22:25.357 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line    900: Feb 17, 2023 09:22:26.728 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  41354: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  41356: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  41358: Feb 17, 2023 12:23:36.362 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  41361: Feb 17, 2023 12:23:36.377 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  41362: Feb 17, 2023 12:23:37.848 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  47326: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  47328: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  47330: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  47333: Feb 17, 2023 15:14:11.829 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  47334: Feb 17, 2023 15:14:12.955 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  53657: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  53659: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  53661: Feb 17, 2023 18:08:55.681 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  53664: Feb 17, 2023 18:08:55.697 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  53665: Feb 17, 2023 18:08:56.667 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  60073: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  60075: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  60077: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  60080: Feb 17, 2023 20:59:02.609 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  60081: Feb 17, 2023 20:59:03.798 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007
    Line  66546: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (WARN) - Timeout while processing events on domain spectroSERVER01; retrying with new request
    Line  66548: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  66550: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - event_sync_time for spectroSERVER01 is 2023-02-07 20:36:35.0
    Line  66553: Feb 17, 2023 23:50:20.761 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - retrieving events on spectroSERVER01 (startTime=2023-02-07 20:36:35.0, endTime=2023-02-17 09:22:24.0)
    Line  66554: Feb 17, 2023 23:50:21.496 (SRM/LandscapeManager/LandscapeThread_18) (SRM_Events) - (DEBUG) - Number of EVENTS retrieved for spectroSERVER01 is 10007

Resolution

 

In order to overcome this situation, SRM needs to fetch events which were generated after 2023-02-07 20:36:35.0 in the above example, hence we need to fetch more events in the single.

So change the threshold limit from 10007 to 50000 .

Do the following on the affected landscape to increase the record search limit: 

  1. stop archive manger. 

  2. edit $SPECROOT\SS\DDM\.configrc 

  3. add record_search_limit=50000 and save it. 

  4. Restart archive manager.

 

Powered by Wolken Software