Java agent is showing excess GC Heap utilization by at least 2 GB for "GC Heap: Bytes in Use" when compared to actual utilization reported by IBM ISA tool.
APM Java agent 9.x, 10.x
The explanation of the difference is as follows:
a. To calculate "GC Heap:Bytes Total" and "GC Heap:Bytes In Use", the Agent uses the following Java API getRuntime() method calls:
"GC Heap:Bytes Total" : java.lang.Runtime.totalMemory()
"GC Heap:Bytes In Use": java.lang.Runtime.totalMemory() - java.lang.Runtime.freeMemory()
Per the Java API documentation: Java™ Platform, Standard Edition 7 API Specification > java.lang > Class Runtime
totalMemory() returns "the total amount of memory currently available for current and future objects, measured in bytes."
freememory() returns "an approximation to the total amount of memory currently available for future allocated objects, measured in bytes."
These calls are taking place every 2.5 seconds and then averaged over the default 15 second reporting interval.
b. Compare the above to the ISA tool (Garbage collection with the IBM Monitoring and Diagnostic Tools for Java - Garbage Collection and Memory Visualizer) which analyses verbose GC logs and they are only generated whenever an actual GC happens. So there is potential for reporting differences due to both the values returned by the runtime functions (e.g. freememory() method returns an approximation) and also the timing of the data collection in GC log versus agent.
In the final analysis for the reported example: