Collection membership is not correct on the Data Aggregator in DX NetOps Performance Management

Products

• CA Performance Management - Usage and Administration

Issue/Introduction

We are updating the rule for a Collection Group in the DX NetOps Performance Management Performance Center (PC) web UI.

This results in 37 devices as members of the Collection instead of the 1000+ device members that were there previously.

This change is displayed in PC correctly. However, when looking at the Collection in the Data Aggregator (DA) Monitored Devices page the Collection still shows 1000+ devices.

The DMService.log shows no error, even when performing a Full synchronization of the DA Data Source after enabling debug for the PUSH phase of the synchronization.

Cause

When a Full DA synchronization is run before the PC side completes the changes on its side behind the scenes to pass them to the DA, this breaks the process. The end result is no change to the Collection membership on the DA side.

At that point further Full DA synchronization cycles will pass the membership from PC to DA but only for the active members. It won't send the necessary Collection Group membership deletes to the DA for removal.

That means in this scenario it will send the correct 37 members of the Collection and the DA will ensure those items are part of the Collection. But the DA won't recognize that it has many more items as members of the Collection that it needs to delete.

Environment

All supported DX NetOps Performance Management releases

Resolution

Once this issue has transpired, there are a few options to resolve it. Listed from most difficult to least difficult we have the following options.

1. The most complex option would require engineering input and intervention. It would entail:
   1. Stop the DA.
   2. Manually clean up Collection membership in the Data Repository DB via commands specific for the given scenario.
   3. Start DA.
2. The second option is simpler but still less desirable as it's not a 'quick' fix for the problem. It would entail:
   1. Resetting the Collection on the PC side back to the original 1000+ item membership.
   2. Wait for that change to be set on the PC side waiting (guesstimate) 20-40 minutes.
   3. Reset the Collection in PC back to the new target of just 37 items as members.
   4. Wait until the DA side catches up and the Collection seen there matches the new membership count.
3. The least complex option is the third option. It's also the fastest path to resolution. It would entail:
   1. Create a new Collection with the correct target membership of 37 items.
   2. Confirm the new Collection shows properly on the DA side.
   3. Create the same Monitoring Profile to Collection associations that the old Collection uses.
      • NOTE: This is key too avoid a data gap and ensures the items are polled continuously.
   4. Delete the old broken Collection from PC.
   5. Confirm the old broken Collection is removed from the DA.
   6. Rename the new Collection to match the old ones name to complete the replacement process.

Additional Information

There is an open story for a change with the engineering team. It will ensure that the deletes are synchronized to the Data Aggregator where they are not done now. This would prevent this scenario from occurring.

There is no known ETA for a release this change would be included in. Please contact your Account Management team for input on that new feature and when it might be available in a future DX NetOps Performance Management release.