SAP : High memory and CPU load after Installing wily /APM agent version 23.x on Application Server .

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

After installing Wily Agent 2023.5.1.13(Build 990013) on the XD1 system, the customer observed high memory and CPU load on application nodes.  After second restart, one node persists with having high memory and CPU load.  

Also server process shutdowns are occurring with errors in the logs ,example below 

"Service process shutting down with exit code [666]; memory allocation error [OutOfMemoryError]"

The heap dump shows the Agent is overloaded with the AgentNameChange event handling:

Most of the heap is occupied by the execution queue for "Agent Execution" thread ("com.wily.util.task.AsynchExecutionQueue"), which contains huge number of entries, most of them are NotifyNameChange and AgentNameChangedExecutableItem. 

 

Environment

Introscope  Agent 2023.5.1.13(Build 990013)
JVM version of managed system XD1 : SAP JVM 8.1.095

Cause

 The root cause is a bug on agent log re-naming introduced when we switched our logging lib from log4j to logback.

 

Resolution

The workaround is set the below in IntroscopeAgent.profile file 

introscope.agent.disableLogFileAutoNaming=true.

This disable log file name auto renaming feature will avoid an endless loop trying to resolve log file name, which keeps consuming memory.

The issue is fixed in which we've fixed in later releases (24.1.1+)