We are seeing instances where UIM metrics that are no longer valid continue to report in APM (via the apmgtw probe) as a static data value. We would expect these metrics to stop reporting, then go gray and delete over time.
Our main concern is to present an accurate state of metric values, and this behavior has generated confusion and mistrust. We need your assistance to remediate this issue.
Our APM environment is at r10.7.0.45, and UIM is at r.9.30 [Build 9.30.1209, Mar 10 2020]
Release : 10.7.0
Component : APM Agents
APM uses the EPAgent httpServer to report the metrics. In this case, this is the expected behavior of repeating same previous value in case of no data coming in.
So the data is kept until new data comes in or the configured aging takes place. APM has metric aging feature to automatically remove metrics reporting same value for a long time, but the default aging threshold is purposely set to very high.
Ideally we should make it configurable on how to handle the missing data scenario, like for the EPAgent stateless plugin we can configure: same, 0, or not reporting. But it would be a ER for EPAgent httpServer. (This would have to be for the Infrastructure Agent)