We moved from using the Prod_APMIA_ASM_* source folders over to the Synthetic source folder in our alerts in early to mid-April, and all was working as expected then.
It appears that around the end of April, all of our ASM related alerts went into Unknown state
Identified the root cause as missing/invalid alert spec update synchronisation issue. When the user modified the alert/metric group in certain order, the new alert spec update is not going to kafka topic and not being updated to metric sorter. so the alert is not getting any metric data and reporting as unknown.
Updated an alert resolution time from 1 minute to 6 minutes to align with ASM metrics reporting every 5 minutes.
An alarm with the “request failed per probe” is now configured