tsdb vm fails to start due to undefined receiver after changes to Healthwatch

Products

• Operations Manager

Issue/Introduction

When making changes to Healthwatch Alertmanager, you encounter an error during Apply Changes indicating that the tsdb vm is not running after the update

Error: 'tsdb/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (0)' is not running after update. Review logs for failed jobs: alertmanager

Alertmanager logs show the following error

ts=2023-10-27T18:19:06.053Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/var/vcap/jobs/alertmanager/config/alertmanager-interpolated.yml err="undefined receiver \"new-receiver\" used in route"

This error indicates that a receiver named new-receiver is not configured to receive alerts.

Resolution

Receivers must be properly defined in Routing rules and Receivers. Correct configuration must match Routing rules and Receiver name.

routes: |
receiver: 'new-receiver'
group_wait: 30s
group_interval: 5m
repeat_interval: 24h
group_by: [...]
inhibit_rules:
receivers:
email_configs: []
pagerduty_configs: new-receiver
slack_configs: []
webhook_configs: