search cancel

APM UMA Google Kubernetes Engine (GKE) Monitor reports incorrect memory usage


Article ID: 215814


Updated On:


DX SaaS DX Application Performance Management


We are trying to set up alerts for container memory usage and found out that AIOPS UMA reports strange numbers.
1) Memory % rounded doesn't show actual memory usage in %:
2) Memory usage is not correct
3) Memory % incorrect value


Release : SAAS



Engineering Analysis on the "Memory Usage" metric:

1. UMA reports the Memory usage inline with "docker stats". Below table shows how docker stats works and the formula used to calculate the memory usage. 


GKE memory usage shows evictable and non-evictable memory

UMA calculates the memory usage by subtracting cache memory from total memory usage. That's why memory usage value was different from GKE total memory 

Docker uses different formulas between versions , as listed below. 

Calculation formula

Memory Usage = Evictable + non-evictable


Evictable = total_inactive_file

Non Evictable = memory.usage_in_bytes- total_inactive_file

Memory Usage = Total Memory Usage - Memory Cache Usage
Memory % = (Total Memory Usage / Memory Limit) * 100)

Docker 1.13 and below :

Memory Usage = Total Memory Usage
Memory % = (Total Memory Usage/ Memory Limit) * 100)

Docker 1.14  till 1.19 :

Memory Usage = Total Memory Usage - Memory Cache Usage
Memory % = (Total Memory Usage - Memory Cache Usage / Memory Limit) * 100)

Docker 1.20+ :

Memory Usage = Total Memory Usage - total_inactive_file
Memory % = (Total Memory Usage - total_inactive_file / Memory Limit) * 100)

Since from Docker 1.20+ , memory usage metric value is equal to "GKE non evicatable" , we will be enhancing the UMA to report a metric that shows "non-evictable" usage value also, as below:

We are adding a new metric called "WorkingSet Memory Usage" and "Memory WorkingSet % (rounded)" that will represents non-evictable memory, where as "Memory Usage (Bytes)" will show total memory (i.e. evictable + non-evictable).

Memory Usage (Bytes)             

Memory % (rounded)                                          

WorkingSet Memory Usage

Memory WorkingSet % (rounded)

[total memory usage]

[(total memory usage/limit) * 100]

[total memory usage-total_inactive_file] = GKE non-evictable value

[total memory usage-total_inactive_file/limit * 100] = GKE non-evictable in %



To be fixed as part of APM SaaS 21.6 release 


Additional Information