CABI slow reports - poor performance - timeouts

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Problem: When attempting to run either custom Ad Hoc reports, or some of the out-of-the-box CABI reports such as the Group Trend report, the performance is very poor.  The reports are slow to populate, and sometimes they time out without ever completing. 

Environment

Release : 20.3

Component : UIM - CABI

Cause

The out-of-the-box settings for cabi and wasp may need tweaking for some environments in order to provide peak performance.

Resolution

There are a number of settings changes within UIM and within CABI which can be tweaked and tuned in order to provide better performance.

It is recommended to change only one of these at a time and evaluate the performance after each change in order to determine the impact.

Memory Settings

The first thing that usually makes a difference is to increase the memory settings from the defaults for both the cabi and wasp probes (on the CABI robot.)  

These settings can be changed in the .cfg or using Raw Configure.  They will be located under <startup> -> <opt> under each probe.

The key/value to look for is:

java_mem_max

We recommend using the same value for both probes here as this seems to provide the best performance.

A decent starting point would be:

java_mem_max = -Xmx6g

In some cases the wasp may already bet set to 6g and you will only need to change it in cabi.

This is only a starting point - generally, higher is better in terms of performance, and in larger environments you may need to set each probe to 12g or higher.

It is important to ensure that you have enough free RAM on the Operating System to accomodate any changes you make and we recommend leaving 4GB free for the OS itself if possible; in other words, if the OS has 32 gigs of RAM available, you could allocate 14 gigs to CABI, and 14 to the wasp probe, leaving 4gb free.

Some larger customers have given as much as 24g to each probe (on a system with 64g available) with excellent results.

 

Virtualizer Settings

After increasing the memory, the next thing to change is to tweak the Virtualizer settings within CABI. This must be done in order to take advantage of the increased memory from the previous step.

The virtualizer is a component which uses virtual memory (on disk) to generate reports in order to avoid OutOfMemory exceptions, but it can be somewhat slow for larger datasets, and it can help to force larger reports to be generated in memory entirely (assuming enough has been given to the probes - it would be wise to monitor the memory usage after making changes here while running reports to ensure you have allocated a sufficient amount.)

To change these settings you will alter a file on the CABI robot (take a backup first!) located at:

$UIM_HOME/nimsoft/probes/service/wasp/webapps/cabijs/WEB-INF/applicationContext.xml

Within this file search for the following section (it appears around line 1316):

 <!--report virtualizers-->
  <bean id="fileVirtualizerFactory" class="com.jaspersoft.jasperserver.api.engine.common.service.impl.FileVirtualizerFactory">
    <property name="maxSize" value="300"/>
    <property name="directory" value="${java.io.tmpdir}"/>
  </bean>

"300" represents the maximum number of pages that can be contained in memory; to force all reports to run in memory, set this to a very high number, e.g. 999999.

Save the file and then restart the cabi robot.

Crosstab Limits

After changing these settings, when attempting to run larger reports, you may receive an error that says "Crosstab bucket limit exceeded."  This can be prevented/corrected as follows:

• stop the wasp probe and cabi probe on the CABI robot
• take a backup of, and then edit, the following file:  $UIM_HOME/nimsoft/probes/service/wasp/webapps/cabijs/WEB-INF/classes/jasperreports.properties
• in this file, locate the following:
  
  

  #limit the crosstab bucket/measure count to prevent out of memory errors
  net.sf.jasperreports.crosstab.bucket.measure.limit=100000

   

• The 100000 represents the maximum number of rows a report can return.  Raising this higher will allow longer reports to run at the expense of using more memory; as a point of reference, one customer increased this to 20000000 (twenty million) and found that the largest reports used around 12gb of RAM on the cabi probe.  Another customer saw usage as high as 28gb with the same settings.  This will depend on the number of metrics/devices and the length of time reported on, so it would be wise to monitor the memory usage to determine how high you can go.

 

Report Design

For custom reports , when you initially create the report and fill out the Input Controls, you will be prompted with a choice between Raw Data and Default Aggregation:

 

 

For larger reports it is important to choose "Default Aggregation" to prevent overwhelming the report with individual samples.

Default Aggregation means that instead of reporting every individual sample, the samples are averaged by every 15 minutes and then these data points are used to draw the graphs/populate the tables.

This is more appropriate for running large group reports or reports that cover a long time period (1 week or more.)

If a problem is noted, and a more granular view is needed, then you can run a single-device report for the desired metric and use Raw Data to get a more granular view.