Spectrum Global Collections missing from Performance Management

Products

CA Infrastructure Management CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

Known issue where during full synchronization of the Spectrum Data Source (DS) in Performance Management (PM) times out, or throws an exception, causing Global Collections (GC) to be removed from Performance Management. When the DS next completes a sync cycle without interruption, the Global Collections are synced again.

The issue may occur without being noticed, and the Global Collection Inventory will look normal. Behind the scenes, in the database, those GCs now have a new ID.

The impact is often seen in Group Rules and Views where the GC is the fixed Context of the View. Those configurations use the ID of the GC in the DB. When it needs to show the name of the GC it references a different table to provide the name. In that table, due to this issue, that ID is no longer present. The GC is in the table but with a new ID due to this known issue.

At this point the Group Rule and Views affected are referencing a GC ID that is no longer present.

Environment

All supported Performance Management releases

Cause

The cause of this issue is a full sync from PM to Spectrum data source interrupted by an error during the pull of groups from Spectrum.

Any error on either the PM or Spectrum OC side can trigger the problem.

Full synchronizations are only launched upon user request via the UI, or data source restarts, being an OC tomcat restart in this case.

The timing of what happened here was OC restarted which triggered the full sync from PM. Then some error interrupted the full sync in that critical groups pull phase.

Resolution

To resolve the impacts, each affected Group Rule or View will need to be edited to reference the newly synced instance of the target GC so the new GC ID is set in the configuration.

There isn't a way to guarantee this won't happen again without upgrading to the r3.7.11 release which includes new code to prevent this problem.

That release was made public early May 2020.

This specific issue is listed in the r3.7.11 Fixed Issues doc page.

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/performance-management/3-7/release-notes/fixed-issues.html

On that page we see:

Symptom: When an error happens in the middle of a Full Pull data source stage, it can result in all items not sent by the data source to be removed from the Performance Center inventory. This could result in new item ids being assigned to them if they only come from a single data source like Spectrum Global Collections.

Resolution: Updated the PC Full Pull synchronization logic to only mark items, not sent by the data source, after the stage is complete. Example: This should stop Spectrum Global Collections from getting assigned a new PC item id, if there is an error during Full pull for groups.

(3.7.11, DE437034, 01142425)

Additional Information

Unfortunately there is no way to determine, after the issue is observed, what the name of the GC was to reset it without a MySql netqosportal DB backup from before the problem start.

If a database backup is available, it can be loaded into a support lab to provide a list of GC names and IDs to use in resolving this.

First determine the Data Source ID for the Spectrum Data Source.

select SourceID,ConsoleName,ConsoleHost,Host from data_sources2 where ConsoleName like '%Spectrum%'\G;

Now run the following using the Spectrum Data Source SourceID value.

select LocalID,ItemID,ItemDesc,ItemName from dst_group where SourceID=<ID>;

That will output entries such as the following example for a single GC.

mysql> select LocalID,ItemID,ItemDesc,ItemName from dst_group where ItemID=24973572;

+----------------------------------+----------+-------------------------+--------------------------+

+----------------------------------+----------+-------------------------+--------------------------+

+----------------------------------+----------+-------------------------+--------------------------+

The columns are defined as:

LocalID: The GC ID it's known as in Spectrum within it's databases
ItemID: The ID it's known as in the Performance Management netqosportal MySql database.
ItemDesc: The name of the GC as it's seen in Spectrum
ItemName: The name of the GC assigned in Performance Management netqosportal MySql database.

When viewing a given PM Collection, Group or View/Report where it presents an ID number instead of the expected GC name, look up the ID in the full list output from dst_group and reset the relationship to the GC again.

In the example of a PM Collection Group or regular Group, in the Rules tab where the ID is seen, resetting it to the correct GC name should resolve the issue for that group. Repeat for others.