Dashboard Views in DX NetOps Performance Management return Query failed errors

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

We have scheduled custom Dashboards from Performance Center to send reports via email.

The PDF report received has Views that show failure and reference a Query ID for log files. For example some Views show this instead of the expected graph.

When reviewing the Data Aggregator karaf.log file (default path /opt/IMDataAggregator/apache-karaf-<version>/data/log) we see errors that reference the following Vertica error.

[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool scheduled_short_rib_processing [Request Too Large:Memory(KB) Exceeded

Environment

All supported DX NetOps Performance Management releases

Cause

Insufficient resources on the Data Repository cluster nodes.

Resolution

These errors and report failures are rooted in the Data Repository (DR) DB system. The DR DB in the environment this was reported in was found to be severely under resourced.

• There should be 64 GB RAM per node with 3 nodes.
  • This system has just two nodes
  • Each node has 16 GB RAM
• There should be 8 CPU cores at 2.60 GHz per node with 3 nodes.
  • The two nodes have the correct minimum number of CPU cores at 8 each.
  • They are slower than the required speed showing 2.20 GHz vs the required 2.60 GHz.

Unfortunately there isn't any tuning that we could recommend to help resolve these report failures. Any tuning performed would just take the limited resources available away from other things resulting in issues in other areas.

Only two things would resolve these report failures in the presence of a severely under resourced system like this. The first is a maybe and isn't a guaranteed solution long term. The report resulting from the first option, by the time it's narrowed down to a request size that doesn't hit resource failures, may not be of much value. The second is a guaranteed solution. It would resolve this problem though it may not be the easiest solution to implement.

1. Create a significantly smaller set of items and metrics to run the report against. That may help to minimize the size of the data response so it doesn't hit these resource limitation issues.
2. Go through the DX NetOps Performance Management Sizing tool. Ensure the environment meets the minimum requirements.
   • The tool is located here: DX NetOps Performance Management Sizing tool
   • Open it to see the basic minimum requirements.
   • Once opened enter the number of Devices being managed by your environment. Recalculate the numbers to get a sense of what to expect for required resources.