Grafana does not show any metrics after a certificate rotation

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

After a CA and leaf certificate rotation, the Grafana UI fails to show any metrics.

The errors in the Grafana UI show something similar to the following:

Status: 500. Message: Get "https://q-s3.tsdb.example.p-healthwatch2-dbeacf25470a4c87a82e.bosh:4450/api/v1/query?query=increase%28tas_sli_failures_total%5B5m%5D%29&time=1759686463.935": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Pivotal")

Environment

Tanzu Platform Cloud Foundry

Healthwatch

Cause

It appears that the "prometheus" row in the data_source table is not getting updated with the new certificate data after a certificate rotation is completed.

Resolution

The following steps can be ran to resolve the issue.  Feel free to reach out to Tanzu Support, via a support request, if you are not comfortable running these steps (as these include a DB delete command).

1. Use bosh ssh to log into the 'pxc' VM.
2. Run 'sudo su vcap'.
3. Run 'mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf' to connect to the mysql DB server and get a DB prompt.
4. At the DB prompt,  run 'use grafana;' to use the grafana DB.
5. Run 'select * from data_source;' to get all rows from the table.  Save this row by copying & pasting into a text file.  Note down the 'id' column value of the row with "prometheus" as the value of the "name" column. 
6. Run 'delete from data_source where id = 1;' to delete the row, where 1 is the 'id' column value retrieved from step 5.
7. Run 'quit' to disconnect from the DB.
8. Then, use bosh ssh to log into the grafana VM.
9. Run 'monit restart grafana'.  During this process, the data_source table would be repopulated with a row that has the updated data.

Afterwards, the Grafana UI should be displaying the metrics again without any errors.