Weblogic application using Java Object Cache (JOC) encounters JVM performance problem due to larger number of wily class weak references being held even after GC of old generation. SQLAgent is being used to monitor SQL activity on the database.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

• SQLAgent is being used and generating extensive Backend metrics.
• The JOC caches data from the database and every 2 hours refreshes it by running a large number of SQL statements which are instrumented by the agent,
• 10Gb heap dump indicated SybPreparedStatement instances occupied about 1 GB memory.
• The agent holds above objects in its weak references and also some strong references (latter only 10Mb)
• All the weak references are eligible for Garbage Collection, so the agent should not be causing the reported performance problem.

1. Oracle Support found that customer had set "-XXgcTrigger=10" which meant that full GC would not occur until the heap was 90% full. Oracle recommended increasing the value of "-XXgcTrigger" to enable full GC to occur earlier or remove it altogether to allow system to automatically find the optimum value: http://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm#BABFCBCI
2. As alternative options CA recommended:
   
   
   1. Disable SQL Agent. However this will remove the ability to collect metrics on all SQL transactions.
   2. Use different JDBC driver for the JOC part of the application and configure APM not to monitor that driver. In that way metrics can still be gathered for the non-JOC SQL transactions.

Environment

Release:
Component: APMAGT