RabbitMQ dashboard in Healthwatch 2.X not working and shows "execution: found duplicate series for the match group (instance=..."

Products

• VMware RabbitMQ

Issue/Introduction

"sum(rabbitmq_connections * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\", namespace=\"$namespace\"})",

When accessing the RabbitMQ dashboard in Healthwatch 2.X, the charts are showing N/A and an exclamation mark is present when you hover over it.

It also shows a red popup window with a message that starts with "execution: found duplicate series for the match group (instance=...".

Installing the Healthwatch Exporter for the Tanzu Application Service (TAS) tile along with the Healthwatch v2.X tile results in duplicated metrics. This is because Healthwatch scrapes the metrics from both the Prometheus port of the RabbitMQ service instance and from the Loggregator system of TAS.

For more information, refer to Duplicated RabbitMQ server metrics when both the Healthwatch Exporter for TAS and the Healthwatch tile are installed.

Environment

Product Version: 2.0

Resolution

There are three workarounds for this issue:

1. Preferred method: Disable Loggregator metrics in RabbitMQ Tile. For more informaiton, refer to Configure the Metrics Polling Interval, by adding a "-1" in the Metrics polling interval field in the RabbitMQ Tile. 

Go to Settings tab > Metrics > Metrics polling interval and then Apply Changes. 

2. Edit the Metrics field for each dashboard manually and add job="rabbitmq" to the string. 

Change:

rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"}

To:

rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace", job="rabbitmq"}

3. This has also been observed due to the match group being one of the HW exporter VMs therefore not unique for the match group. To remedy this, update each dashboard manually changing (instance)  to (ip).

Change:

sum(rabbitmq_connections * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"})

To:

sum(rabbitmq_connections * on(ip) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster="$rabbitmq_cluster", namespace="$namespace"})

 

Known Issue

RabbitMQ tile v2.1.6 contains a bug where setting the scape interval to "-1" does not work and sends prom_scraper into a crash loop. This has since been patched starting with RabbitMQ tile v2.1.7+ and v2.2.1+.