AWS monitor reporting java.lang.OutOfMemoryError: Java heap space

Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management

Issue/Introduction

The Infrastructure Agent, AWS Monitor is not working as expected, the below message is reported in the IntroscopeAgent.log

2/13/20 04:31:10 PM CET [ERROR] [IntroscopeAgent.RESTMon] java.lang.OutOfMemoryError: Java heap space
2/13/20 04:31:10 PM CET [ERROR] [IntroscopeAgent.RESTMon] Exception in: eu-west-3~~redshift. Will attempt to re-establish database connection.
java.sql.SQLTransientConnectionException: connection exception: connection failure: java.io.EOFException
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.JDBCPreparedStatement.executeQuery(Unknown Source)
at com.ca.ce.restmon.db.dao.ProfileDefinitionDao.getProfileDefinitionByName(ProfileDefinitionDao.java:598)
at com.ca.ce.restmon.db.DbUtil.getProfileDefinitionByName(DbUtil.java:468)
at com.ca.ce.restmon.core.ProfileHandler.processProfile(ProfileHandler.java:218)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.run(MetricCollector.java:96)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:79)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:1)

Environment

Application Performance Management 10.7, 11.x, 20.2 releases.
APM Agents SaaS

Cause

Infrastructure agent (IA) is by default configured with 512MB that might not be the appropriate for the monitor(s) in use.

You must adjust the IA heapsize based on your tests and requirements.

Resolution

Suggestions:

1. Increase heap size memory: we recommend to configure the Infrastructure Agent for AWS monitor with a minimum of 1 GB to prevent the Out of Memory issue

As per documentation:
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/infrastructure-agent/install-the-infrastructure-agent.html#concept.dita_a3e333bf4a28317f3fe2f7974a211e6d2397b28f_ConfiguretheInfrastructureAgentHeapSize

"
Configure the Infrastructure Agent Heap Size

You can configure the xms and xmx parameters of the JVM in which the Infrastructure Agent runs using either command line parameters or environment variables.

Configure Heap Size Using Command Line Parameters

Run the APMIA install script including min_heap and max_heap parameters:

UNIX: ./apmia-ca-installer.sh install min_heap=<value> max_heap=<value>

Windows: apmia-ca-installer.bat install min_heap <value> max_heap <value>

Mac: ./apmia-ca-installer.sh install min_heap=<value> max_heap=<value>

By default, min_heap is 256 and max_heap is 512. You can use the installer script to update java heap values on start and restart.

Configure Heap Size Using Environment Variables

Alternatively, you can use the following environment variables to set the heap size values. Once configured, the values are picked up by the installation script when you run it.

MIN_HEAP_VAL_IN_MB

MAX_HEAP_VAL_IN_MB

For example on Linux:

$ export MIN_HEAP_VAL_IN_MB=256

$ export MAX_HEAP_VAL_IN_MB=512

Heap size values provided on the command line override the environment variables."

2. Try reducing the amount of metrics being gathered for example, limit the regions and services in the bundle.properties file.

introscope.agent.aws.profiles.aws.schemaList
introscope.agent.aws.profiles.aws.regionList

3. Keep in mind below important note from documentation when configuring the monitor:

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/dx-apm-saas/SaaS/implementing-agents/infrastructure-agent/amazon-web-services-monitoring.html

"Important!
Amazon charges the AWS account that AWS Monitoring uses to monitor various AWS services. We recommend that you consider the billing costs while configuring AWS Monitoring"