search cancel

APM UMA Google Kubernetes Engine (GKE) Monitor reports incorrect memory usage

book

Article ID: 215814

calendar_today

Updated On:

Products

DX SaaS DX Application Performance Management

Issue/Introduction

We are trying to set up alerts for container memory usage and found out that AIOPS UMA reports strange numbers.
 
1) Memory % rounded doesn't show actual memory usage in %:
2) Memory usage is not correct
3) Memory % incorrect value
 
 

Environment

Release : SAAS

Component : APM SAAS ENTERPRISE

Cause

Engineering Analysis on the "Memory Usage" metric:

1. UMA reports the Memory usage inline with "docker stats". Below table shows how docker stats works and the formula used to calculate the memory usage. 

 

GKE memory usage shows evictable and non-evictable memory

UMA calculates the memory usage by subtracting cache memory from total memory usage. That's why memory usage value was different from GKE total memory 

Docker uses different formulas between versions , as listed below. 

Memory
Calculation formula

Memory Usage = Evictable + non-evictable

 

Evictable = total_inactive_file

Non Evictable = memory.usage_in_bytes- total_inactive_file

Memory Usage = Total Memory Usage - Memory Cache Usage
Memory % = (Total Memory Usage / Memory Limit) * 100)

Docker 1.13 and below :

Memory Usage = Total Memory Usage
Memory % = (Total Memory Usage/ Memory Limit) * 100)

Docker 1.14  till 1.19 :

Memory Usage = Total Memory Usage - Memory Cache Usage
Memory % = (Total Memory Usage - Memory Cache Usage / Memory Limit) * 100)

Docker 1.20+ :

Memory Usage = Total Memory Usage - total_inactive_file
Memory % = (Total Memory Usage - total_inactive_file / Memory Limit) * 100)

Since from Docker 1.20+ , memory usage metric value is equal to "GKE non evicatable" , we will be enhancing the UMA to report a metric that shows "non-evictable" usage value also, as below:

We are adding a new metric called "WorkingSet Memory Usage" and "Memory WorkingSet % (rounded)" that will represents non-evictable memory, where as "Memory Usage (Bytes)" will show total memory (i.e. evictable + non-evictable).

Memory Usage (Bytes)             

Memory % (rounded)                                          

WorkingSet Memory Usage

Memory WorkingSet % (rounded)

[total memory usage]

[(total memory usage/limit) * 100]

[total memory usage-total_inactive_file] = GKE non-evictable value

[total memory usage-total_inactive_file/limit * 100] = GKE non-evictable in %

 

Resolution

To be fixed as part of APM SaaS 21.6 release 

 

Additional Information

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-saas/SaaS/DX-APM-Saas-Whats-New.html

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-saas/SaaS/implementing-agents/Universal-Monitoring-Agent/Container-Monitoring/Pod-Performance-Data.html#concept.dita_4087a02b-3b2c-4f23-982e-c3e0fb774264_KubernetesandOpenShiftMetrics