POD Memory Utilization reported by UMA Agent

Products

• DX APM SaaS

Issue/Introduction

kubectl top PODs and metric values from the UMA seem do not match. 

Is there a way to get the values from k8s cluster that matches what the UMA agent is reporting?

Wants to understanding how the UMA agent calculates memory utilization for the POD. 

Found a discrepancy where the memory utilization reported for the POD is below 40% when checking with "kubectl top PODs" command, but UMA Agent shows utilization of 70% for Memory % (rounded).

There are instances where APM shows utilization reaching 100%, yet the applications team has not observed any restarts for the POD.

Resolution

Suggested to compare the output of command "kubectl top pods" with following UMA agent values:

SuperDomain|<UMA cluster>|<node>|Kubernetes Agent|Kubernetes|Namespaces|<namespace>|Pods|<pod name>:Memory Working Set

AND

SuperDomain|<UMA cluster>|<node>|Kubernetes Agent|Kubernetes|Namespaces|<namespace>|Pods|<pod name>:Memory Working Set %

These above two values should be more comparable with "kubectl top pods" and kubernetes dashboards.

The UMA agent gets memory metrics via API call to the Kubernetes cluster.

Memory % (rounded) is not the same as above. The UMA agent calculate as follows:

Memory % (rounded)
Memory utilization by the pod over time. This percentage is Memory Usage (Bytes) divided by Memory Limit (Bytes).

Memory Usage (Bytes) includes memory POD is using + all other memory use by POD which also include the cache memory which is not included with top command. It is not the same as "kubectl top pods".