How to configure Alertmanager in Healthwatch2 to sent an alert to multiple receivers

Products

• Operations Manager

Issue/Introduction

Sometimes users may have configured multiple receivers for Alertmanager on Healthwatch2 Settings page. And it's expected that an alert could be sent to all the configured receivers at the same time. This article shows one way to achieve the goal.

Environment

Product Version: 2.10

Resolution

As described in Prometheus documentation:
A route block defines a node in a routing tree and its children.  If continue is true on a matching node, the alert will continue matching against subsequent siblings. 

So the continue statement can be utilized to route one alert to multiple receivers simultaneously. Routing rules can be configured on Alertmanager page of Healthwatch2 Settings. Suppose both Slack and Email receivers are configured, then use the following example of routing rules to send an alert to both Slack and Email receivers at the same time.

route:
  receiver: slack
  group_interval: 5m
  repeat_interval: 4h
  group_by: [cluster, alertname]
  routes:
     - receiver: slack
       continue: true
     - receiver: email

Also as demonstrated by image blow. 


If it doesn't work as expected, review /var/vcap/sys/log/alertmanager/alertmanager.stderr.log file on tsdb instances of Healthwatch2 deployment for any errors.