What is the difference between DX-APM Java agent metrics “GC Monitor:Percentage of Java Heap Used” and “JMX|JVM|Memory|Heap:Percentage of Max Heap Currently Used”

Products

• DX SaaS

Issue/Introduction

Java agent Release 21.11.0.20 (Build 990020) running in PCF (Pivotal Cloud Foundry) environment, report different values for the metrics:
GC Monitor:Percentage of Java Heap Used
JMX|JVM|Memory|Heap:Percentage of Max Heap Currently Used
as shown in following screen shot:

When values differ the GC Monitor one appears lower though differences are not large.  I notice the 'count' on values varies which could explain these are both consuming same 'source' in JMX but at different points in time or aggregating different number of values so result in observed discrepancies.  The value difference seems higher when there is a difference in count, but not sure that always holds true.

Questions:
1) should these metrics report same value based on same count?

2) if not, which is more correct?

3) given that "GC Monitor:Percentage of Java Heap Used" is very long standing Java agent metric and easier to use in metric groupings as no pipe characters to escape in metric specifier regex, is there ever any advantage to use "JMX|JVM|Memory|Heap:Percentage of Max Heap Currently Used"?

 

Environment

Release : SAAS

Resolution

While both metrics are gathered and computed via JMX API calls, they are not using the same data set from the same invocations of JMX API calls.  

Their individual agent modules would invoke the JMX API calls at different times and compute their metric values based on separate sets of aggregated data.

Thus, they are more a normalized or average data value over their specific time intervals.  

They can use either metric to help gauge the heap usage at a certain time period, but these metric values may not be the exact heap usage number at that particular point of time.