Data Collectors show data gaps tied to growing heap usage

Products

• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

The System Health Dashboards for Data Collectors (DC) shows heap usage increasing until it potentially reaches 90-100%.  As the problem grows we can see an increase in "dropped poll request" messages in the karaf.log file on the affected DC. Once those messages begin appearing the spread of the data gaps grows until no data is seen from the DC. Only dcmd service stop and restart has helped clear up the issue.

Environment

DX Netops Performance Management 23.3.2 and earlier

Cause

Defect DE577202

Resolution

This issue is fixed in DX NetOps 23.3.3 and later:

Symptom: In large SNMPv3 environments, polling can slow down and back up due to how fast SNMP4j MPv3 cache is used. It is using an inefficient search algorithm.
Resolution: With this fix, the MPv3 cache now includes an additional hash map that uses a different key, and the lookups are now faster. MPv3 cache no longer slows down polling.
(23.3.3, DE577202, 33529643)

Additional Information

Potential Workaround:

Move SNMPv3 polled devices to data collectors if they have capacity and are in the same IP domain that you can move.

Keep in mind that before 23.3.9, this also resets baselines for those devices.