Healthwatch email alerting stuck due to TLS issue with SMTP server

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

The customer configured email alerting rules in Alertmanager through the Healthwatch tile, but they were not receiving any emails.

Environment

Operation manager v3.x

Healthwatch v2.3.x

Cause

TLS is enabled on the Alertmanager side, but not on the SMTP server.

Resolution

First, temporarily enable debug logging on the Healthwatch side by editing the Alertmanager YAML file inside the TSDB VMs.

• Edit "/var/vcap/jobs/alertmanager/config/bpm.yml" and update below line with "debug".

  - "--log.level=debug"

After enabling debug logging, any TLS-related issues during communication with the SMTP server will be visible in "/var/vcap/sys/log/alertmanager/alertmanager.stderr.log". In this case, TLS was not enabled on SMTP server side.

To resolve the issue, we disabled TLS on the Alertmanager side by adding the "require_tls" flag under the email alerting receivers and setting it to "false".

Then do the "apply change" on healthwatch tile. Eventually you will able to see logs related to alert flushing and will email in mail boxes.

 

 

Note: One can use "amtool" CLI inside TSDB VM to trigger manual email alerts. Below is the sample command.

amtool alert add --alertmanager.url="http://localhost:10401” 'alertname=CertificateExpiration' summary="This is a test email alert from amtool."