Unable to see SystemEdge metrics on one of our OpenShift nodes..

Products

• DX SaaS

Issue/Introduction

Unable to see the SystemEdge metrics for this node, and pods on it at not appearing in APM. This behavior seems like we are hitting a clamp for this agent.

This node has the same resources as the other nodes in the cluster.

Can you confirm if we are hitting a clamp or what may be causing these issues?

Restarting the app-container-monitor pod for the node resolves the issue.

Want root cause 

Resolution

Yes, the metric clamp of 100k was hit.

The app-container-monitor (Kubernetes Agent) pod on node was continuously hitting its 100,000 metric clamp limit, which prevented SystemEdge metrics (Availability, System UpTime) from being reported and caused pods to not appear in APM.

Although the Agent Metric Count in UI shows only 16k live metrics, there were probably many more stale ones due to continuous auto-attach scans across hundreds of pods on this dense node.

Restarting the pod resolved the issue by clearing the accumulated stale metric state, allowing SystemEdge metrics to be registered before the registry refilled.