Discrepancy between Bytes In Use and Percentage of Java Heap Used metric values

### Discrepancy between Bytes In Use and Percentage of Java Heap Used metric values

book

calendar_today

#### Products

CA Application Performance Management SaaS

#### Issue/Introduction

Given that GC Heap:Bytes Total is constant, expecting that GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used would vary in the same relative way (as I assumed GC Monitor:Percentage of Java Heap Used was computed from the two GC Heap metrics).

However, this is not the case.  I notice the 'count' for GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used differs.

Does this explain the different values?   Also the aggregated GC Monitor:Percentage of Java Heap Used values over longer time intervals (1 min, 5 min etc) seem to have values higher than values at finer resolutions such as 15 sec.

How are these values aggregated - I would expect simple average.  Could they be weighted averages as the count differ?

#### Environment

Release : SAAS

Component : Integration with APM

#### Resolution

GC Monitor:Percentage of Java Heap Used" metric is calculated with this formula:

(GC Heap:Bytes In Use ÷ GC Heap:Bytes Total) * 100

Solution:

1. "GC Monitor:Percentage of Java Heap Used" metric is calculated with this formula:

(GC Heap:Bytes In Use ÷ GC Heap:Bytes Total) * 100

2. The above metric when calculated should be approximately equal to the average obtained by doing the below calculation for each of the memory pools(for example: old space, nursery, class memory in case of Jrockit JVM).

(Amount of Space used ÷ Current capacity) * 100

Basically these 2 metrics (GC Heap:Bytes In Use and GC Monitor:Percentage of Java Heap Used):

1. are from 2 different sources, although theoretically they should be equivalent.

- metrics under "GC Heap" were queried from JVM Runtime.

- metrics under "GC Monitor" were queried/calculated from JMX.

2. have different publish frequency.

- "GC Heap:Bytes In Use" has polling frequency of 2.5s, therefore 24 data points in 1 minute.

- "GC Monitor:Percentage of Java Heap Used" is calculated every 7.5s, therefore 8 data points in 1 minute.

3. have different metric type, so different way of aggregation.

- "GC Heap:Bytes In Use" is of "LongFluctuatingCounter", which uses the last data point value as aggregation result.

- "GC Monitor:Percentage of Java Heap Used" is of "IntegerAverage", which aggregates by averaging.

- We query either JVM/JMX for memory/heap usage as whole, so it's not our decision whether Metaspace was included in or not.

For "GC Monitor:Percentage of Java Heap Used" calculation, Agent gets both bytes in use and total value directly from JMX:

java.lang.management.MemoryMXBean.getHeapMemoryUsage()

.getUsed() is the bytes in use.

.getMax() (if it's >0), or .getCommitted(), is the bytes total.

Also confirmed, we do normalize the calculation result of above to avoid value > 100.