Alertmanager no longer sending alerts after Healthwatch upgrade

Products

• VMware Tanzu Application Service

Issue/Introduction

• After upgrading Healthwatch from versions prior to 2.2.9 to later versions, custom alerts configured in Alert Manager are no longer triggered.
• New receiver configurations not working in Healthwatch Alert Manager when using configurations from older environments.
• After adding new team DLs, alerts are not sent.
• The custom alerts are configured with origin="bosh-system-metrics-forwarder".

Environment

Healthwatch 2.2.9+

Cause

In Healthwatch 2.2.9, there was a breaking change that now requires Healthwatch to use the system-metrics-agent. This is detailed in the Release notes for Healthwatch 2.2.9 on page 13:

 

"[Breaking Change] Healthwatch now requires the use of system-metrics-agent processes to gather “system” metrics from BOSH deployed VMs. Please make sure to enable Enable System Metrics in the Director Config tab of the BOSH Director tile. Ensure that Apply Changes is run on all tiles and that all VMs deployed via service brokers are upgraded prior to upgrading Healthwatch. The impact of not performing this action is that Healthwatch dashboards will fail to populate with metrics about VM health, cpu, memory, disk and other statistics. Alerts may also be affected. Any custom dashboards that query “system” metrics with an origin label value of bosh-system-metrics-forwarder will need to be updated to use system-metrics-agent as the origin label value."

Resolution

Modify the custom alerts in Healthwatch tile under Alertmanager to pull from origin="system_metrics_agent" to correctly query metrics in order to trigger alerts.