search cancel

Alarm message text contents causing log alerts to not be shown in DX Dashboard

book

Article ID: 229606

calendar_today

Updated On:

Products

DX Operational Intelligence

Issue/Introduction

We ran into a problem where for our production alarms, if we put the wrong text (variables) in an alarm message, the log alerts stop triggering even though the log lines still exist that they should trigger on.  i.e. the problem is occurring, but the alarm stops showing up on the dashboard just due to the alarm message text content change.

The alarm message content of:

STXXXXX0001 No Input eml msgs {{ctx.results.0.hits.hits[0]._source.host}} {{ctx.results.0.hits.hits.0._source.cloud.project.id}} {{ctx.results.0.hits.hits.0._source.message}}

However, something in this alarm message text with the variables that we had copied from other example alerts caused the back end to not like it or display the logs anymore:

{{ctx.results.0.hits.hits[0]._source.host}} {{ctx.results.0.hits.hits.0._source.cloud.project.id}} {{ctx.results.0.hits.hits.0._source.message}}

When we took all that out and just left this in:

STXXXXX0001 No Input eml msgs

The logs show up again. The issue was that on the alarm definition side, something in the above was causing things to not work when it tried to pull for matching logs.

Now that those extra pieces are out, my alarms and log alerts work again. 

So if you could please explain the allowed text for alarm messages so we can sanity check all of our production alarms that would be helpful. i.e. the variable values we tried to use here were copied from other active alerts, so this may be impacting other alarms currently in production. This may be causing other alarms to not show up on the production dashboards and cause customer outages.

Environment

Release : 21.3

Component : CA DOI LOG ANALYTICS

Resolution

When accessing the index of the element in the resulting array don't use the below syntax

{{ctx.results.0.hits.hits[0]._source.host}}
i.e. don't use brackets.

rather use 
{{ctx.results.0.hits.hits.0._source.host}}