Wavefront/DXOE Alert Notifications are not delivered to Google Chat Webhooks.

Products

• DX OpenExplore
• Insights SaaS
• Observability

Issue/Introduction

Custom Alert Targets fail to deliver Webhook notifications to Google Chat Space.  Slack or PagerDuty Webhooks deliver successfully from Wavefront or DX OpenExplore.

Cause

Functionality of sending notifications to Google Chat from Wavefront or DX OpenExplore is not currently supported.

Resolution

It is recommended to use a supported notification method for reliable delivery of alert messages from your Wavefront or DX OpenExplore environment. 

If you choose to use Google Chat APIs for Alert notification the below steps were provided as best-effort and have been found to deliver Alert Notifications however are not guaranteed. 

1. Create a Webhook in your destination Google Chat Space. 
   1. Example: https://chat.googleapis.com/v1/spaces/<SpaceID>/messages?key=<API_KEY>&token=<TOKEN>
      
      
2. Create a Custom Alert Target with the UI.
   
   
   1. Enter needed information for  Alert Target Name, Description, ect. 
      
      
   2. Modify the out-of-the-box Custom Body Template. 
      
      The below is an example of a working Custom Body Template for Google Chat (Space) Webhook.

{{! https://docs.wavefront.com/webhooks_alert_notification.html#customizing-alert-target-templates }}
{{#setDefaultIterationLimit}}500{{/setDefaultIterationLimit}}
{{#setFailingLimit}}500{{/setFailingLimit}}
{{#setInMaintenanceLimit}}500{{/setInMaintenanceLimit}}
{{#setNewlyFailingLimit}}500{{/setNewlyFailingLimit}}
{{#setRecoveredLimit}}500{{/setRecoveredLimit}}
{{#setAllInAlertQueryLimit}}500{{/setAllInAlertQueryLimit}}
{{#setFilterByKVsLimit}}500{{/setFilterByKVsLimit}}
{{#setContributingKVsLimit}}100{{/setContributingKVsLimit}}

{
  "cardsV2": [{
            "card": {
                "header": {
                   "title": "[{{severity}}] {{name}}",
                   "subtitle": "{{reason}}"
                },
                "sections": [
                    {
                       "header": "Alert Link",
                       "widgets":[{"textParagraph": {"text":"<a href={{{url}}}>{{url}}</a>"}}]
                    },
                    {
                     "header": "Failing Sources",
                       "widgets": [
                          {
                             "textParagraph": {"text": "{{#failingHosts}} {{{.}}}, {{/failingHosts}}"}
                          }
                        ]
                    },
                    {
                     "header": "Newly failing Sources",
                       "widgets": [
                          {
                             "textParagraph": {"text": "{{#newlyFailingHosts}} {{{.}}}, {{/newlyFailingHosts}}{{^newlyFailingHosts}}n/a{{/newlyFailingHosts}}"}
                          }
                        ]
                    },
                    {
                     "header": "Recovered Sources",
                       "widgets": [
                          {
                             "textParagraph": {"text": "{{#recoveredSeries}} {{{.}}}, {{/recoveredSeries}}{{^recoveredSeries}}n/a{{/recoveredSeries}}"}
                          }
                        ]
                    }
                ]
            }
        }]
}

Additional Information

Limitation:

• Test a Custom Alert Target does not work with Google Chat Webhook. 
  • Workaround to confirm delivery of Alert Notification to Google Chat Webhook is to create a test alert with the custom Alert Target.
• Not all Mustache syntax will work when sending to a Google Chat Webhook.

Documentation:

Customizing Alert Notifications 

• We define variables for obtaining information about the alert as a whole, such as the alert ID, timing, severity, and so on.
• Each of these variables is a property unless explicitly described as an iterator.