AWS monitor reporting java.lang.OutOfMemoryError: Java heap space

book

Article ID: 182981

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management

Issue/Introduction

The Infrastructure Agent, AWS Monitor is not working as expected, the below message is reported in the IntroscopeAgent.log

2/13/20 04:31:10 PM CET [ERROR] [IntroscopeAgent.RESTMon] java.lang.OutOfMemoryError: Java heap space
2/13/20 04:31:10 PM CET [ERROR] [IntroscopeAgent.RESTMon] Exception in: eu-west-3~~redshift. Will attempt to re-establish database connection.
java.sql.SQLTransientConnectionException: connection exception: connection failure: java.io.EOFException
 at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
 at org.hsqldb.jdbc.JDBCUtil.sqlException(Unknown Source)
 at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(Unknown Source)
 at org.hsqldb.jdbc.JDBCPreparedStatement.executeQuery(Unknown Source)
 at com.ca.ce.restmon.db.dao.ProfileDefinitionDao.getProfileDefinitionByName(ProfileDefinitionDao.java:598)
 at com.ca.ce.restmon.db.DbUtil.getProfileDefinitionByName(DbUtil.java:468)
 at com.ca.ce.restmon.core.ProfileHandler.processProfile(ProfileHandler.java:218)
 at com.ca.apm.agent.extension.restmon.core.MetricCollector.run(MetricCollector.java:96)
 at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:79)
 at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:1)

Cause

Infrastructure agent (IA) is by default configured with 512MB that might not be the appropriate for the monitor(s) in use. 

You must adjust the IA heapsize based on your tests and requirements.

Environment

Application Performance Management 10.7, 11.x, 20.2 releases.
APM Agents SaaS

Resolution

Suggestions:

1. Increase heap size memory:  we recommend to configure the Infrastructure Agent for AWS monitor with a minimum of 1 GB to prevent the Out of Memory issue

As per documentation:
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/infrastructure-agent/install-the-infrastructure-agent.html#concept.dita_a3e333bf4a28317f3fe2f7974a211e6d2397b28f_ConfiguretheInfrastructureAgentHeapSize
"
Configure the Infrastructure Agent Heap Size
You can configure the xms and xmx parameters of the JVM in which the Infrastructure Agent runs using either command line parameters or environment variables.

Configure Heap Size Using Command Line Parameters
Run the APMIA install script including min_heap and max_heap parameters:

UNIX:  ./apmia-ca-installer.sh install min_heap=<value> max_heap=<value>
Windows: apmia-ca-installer.bat install min_heap <value> max_heap <value>
Mac: ./apmia-ca-installer.sh install min_heap=<value> max_heap=<value>

By default, min_heap is 256 and max_heap is 512. You can use the installer script to update java heap values on start and restart.

Configure Heap Size Using Environment Variables

Alternatively, you can use the following environment variables to set the heap size values. Once configured, the values are picked up by the installation script when you run it.

MIN_HEAP_VAL_IN_MB
MAX_HEAP_VAL_IN_MB

For example on Linux:
$ export MIN_HEAP_VAL_IN_MB=256
$ export MAX_HEAP_VAL_IN_MB=512

Heap size values provided on the command line override the environment variables."

2. Try reducing the amount of metrics being gathered for example, limit the regions and services in the bundle.properties file.

introscope.agent.aws.profiles.aws.schemaList 
introscope.agent.aws.profiles.aws.regionList 

3. Keep in mind below important note from documentation when configuring the monitor:

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/dx-apm-saas/SaaS/implementing-agents/infrastructure-agent/amazon-web-services-monitoring.html

"Important!
Amazon charges the AWS account that AWS Monitoring uses to monitor various AWS services. We recommend that you consider the billing costs while configuring AWS Monitoring"