We have created APM Alerts against the StatefulSet Ready Metric and the Alert always remains in an Unknown state. Here is the configuration for the Alert.
This has been configured for some time and no change. We have tried adjusting the thresholds and comparison operators, but still shows as unknown. We see that both (metric grouping preview and alert preview) are showing the metric values.
Release : SAAS
Component : Introscope
It seems like following is the root cause why the alerts are not working.
In the broken cases the metric grouping -> metric specifier is
Kubernetes\|Namespaces\|sexxxxxxlse\-prod\|Statefulsets\|raxxxxxxxxxxxx1a:Ready
In the working cases the metric grouping -> metric specifier is
Kubernetes\|Namespaces\|sexxxxxxlse\-prod\|StatefulSets\|raxxxxxxxxxxxx1a:Ready
The difference being the "S" in StatefulSets is capitalized in the working scenario.
It seems that the metric preview must be doing a case insensitive regex match while the actual alert is doing a case sensitive regex match.