Discrepancy between Bytes In Use and Percentage of Java Heap Used metric values

Products

• CA Application Performance Management SaaS

Issue/Introduction

Given that GC Heap:Bytes Total is constant, expecting that GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used would vary in the same relative way (as I assumed GC Monitor:Percentage of Java Heap Used was computed from the two GC Heap metrics). 

However, this is not the case.  I notice the 'count' for GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used differs.

Does this explain the different values?   Also the aggregated GC Monitor:Percentage of Java Heap Used values over longer time intervals (1 min, 5 min etc) seem to have values higher than values at finer resolutions such as 15 sec. 

How are these values aggregated - I would expect simple average.  Could they be weighted averages as the count differ?

Environment

Release : SAAS

Component : Integration with APM

Resolution

GC Monitor:Percentage of Java Heap Used" metric is calculated with this formula:

(GC Heap:Bytes In Use ÷ GC Heap:Bytes Total) * 100

Solution:

1. "GC Monitor:Percentage of Java Heap Used" metric is calculated with this formula: 
   
   (GC Heap:Bytes In Use ÷ GC Heap:Bytes Total) * 100 
   
   
2. The above metric when calculated should be approximately equal to the average obtained by doing the below calculation for each of the memory pools(for example: old space, nursery, class memory in case of Jrockit JVM). 
   
   (Amount of Space used ÷ Current capacity) * 100

Basically these 2 metrics (GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used):

1. are from 2 different sources, although theoretically they should be equivalent.

- metrics under "GC Heap" were queried from JVM Runtime.

- metrics under "GC Monitor" were queried/calculated from JMX.

2. have different publish frequency.

- "GC Heap:Bytes In Use" has polling frequency of 2.5s, therefore 24 data points in 1 minute.

- "GC Monitor:Percentage of Java Heap Used" is calculated every 7.5s, therefore 8 data points in 1 minute.

3. have different metric type, so different way of aggregation.

- "GC Heap:Bytes In Use" is of "LongFluctuatingCounter", which uses the last data point value as aggregation result.

- "GC Monitor:Percentage of Java Heap Used" is of "IntegerAverage", which aggregates by averaging.

 

- We query either JVM/JMX for memory/heap usage as whole, so it's not our decision whether Metaspace was included in or not.

 

For "GC Monitor:Percentage of Java Heap Used" calculation, Agent gets both bytes in use and total value directly from JMX:

java.lang.management.MemoryMXBean.getHeapMemoryUsage()

.getUsed() is the bytes in use.

.getMax() (if it's >0), or .getCommitted(), is the bytes total.

 

Also confirmed, we do normalize the calculation result of above to avoid value > 100.