Experience of strange behaviour UMA v.18.104.22.168:
The Openshift Agent for the service that was reporting all the metric, the last replica of the pods was not captured yet and not connected with node.
Only the pods (that are running) under IA are visible
but not under the node, like this previous pod replica:
There was an issue with the clusterinfo pod which could not be contacted.
The Java autoattach module gets some pod related data from clusterInfo before attaching agents. Since clusterinfo was not returning any data attach module was failing.
The clusterinfo was not responding because of known issues in this older release that caused this to happen because of exceptions in clusterinfo internal threads, such as this:
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
The exceptions seen in clusterinfo log have been fixed in latest releases.
Release : 10.7.0
Component : APM Agents
The suggestion was to redeploy the clusterinfo pod with a liveness check
An attached version of clusterinfo.yaml for 20.4 is provided
As we have asked the user to add the livenessprobe, if the issue happens again in their env pod will get restarted automatically.