AlertManager is incompatiable between Healthwatch v2.x

Products

Operations Manager

Issue/Introduction

Ops Manager v2.10.19 deploys BOSH Director v271.11.0.

Ruby 3 is packaged with the Director starting with Director v271.10.0.

In Ruby 3, URI.escape is deprecated. For more information about URI.escape being depreciated, refer to Can't call escape method on Ruby 3.0.0 #14.

Healthwatch v2.1.4 and earlier (not Healthwatch v1.x) packages a file called alerting.rules.yml.erb that contains a call to URI.escape. This call is only made if there are alerting rules configured in the Alertmanager tab in Healthwatch.

Upgrading to Ops Manager v2.10.19 with alerting rules configured in Healthwatch will cause an error when rendering templates for Healthwatch deployment that looks like the following:

===== 2021-10-12 23:18:03 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=IP.IP.IP.IP --deployment=p-healthwatch2-GUID deploy --no-redact /var/tempest/workspaces/default/deployments/p-healthwatch2-GUID.yml"
Using environment 'IP.IP.IP.IP' as client 'ops_manager'

Using deployment 'p-healthwatch2-GUID'

Task 2496091

Task 2496091 | 23:18:05 | Preparing deployment: Preparing deployment (00:00:04)
Updating deployment:
Expected task '2496091' to succeed but state is 'error'
Exit code 1
Task 2496091 | 23:18:09 | Preparing deployment: Rendering templates (00:00:09)
L Error: Unable to render instance groups for deployment. Errors are:
- Unable to render jobs for instance group 'tsdb'. Errors are:
- Unable to render templates for job 'prometheus'. Errors are:
- Error filling in template 'alerting.rules.yml.erb' (line 33: undefined method `escape' for URI:Module)
Task 2496091 | 23:18:18 | Error: Unable to render instance groups for deployment. Errors are:
- Unable to render jobs for instance group 'tsdb'. Errors are:
- Unable to render templates for job 'prometheus'. Errors are:
- Error filling in template 'alerting.rules.yml.erb' (line 33: undefined method `escape' for URI:Module)

Task 2496091 Started Tue Oct 12 23:18:05 UTC 2021
Task 2496091 Finished Tue Oct 12 23:18:18 UTC 2021
Task 2496091 Duration 00:00:13
Task 2496091 error

Environment

Product Version: 2.10

Resolution

Since the call to URI.escape is in the template before BOSH interpolates it with Ruby, there is no permanent solution in place.

Current Recommendations

The preferred recommendation is to upgrade Healthwatch to v2.1.5+ as this issue is patched starting in that version.

If you can not upgrade Healthwatch to v2.1.5+ then follow the below instructions.

Note: Since this error only occurs if alerting rules are configured in the Alertmanager tab in Healthwatch in combination with Ops Manager v2.10.19 and Healthwatch v2.1.4, these recommendations are based on whether there are alerting rules configured in your Healthwatch.

If you have alerting rules configured in Healthwatch, but have not upgraded to Ops Manager v2.10.19, then either of the following can be done to work around this issue:

Wait for the patched Healthwatch to release before upgrading to Ops Manager v2.10.19.
Upgrade to Ops Manager v2.10.19, but exclude Healthwatch from the Apply Changes until the patched release is staged and ready to update.
Upgrade to Ops Manager v2.10.19 but remove all alerting rules in the Alertmanager tab in the Healthwatch tile and Apply Changes. The deploy should be successful, however alerting will not work until updating to the patched version and re-adding the rules.

If you do not have alerting rules configured in Healthwatch tile, then your Healthwatch deployment should not be impacted from this Ops Manager upgrade.