Certificate expiration warning still shown on Grafana post successful certificate rotation

Products

• VMware Tanzu Application Service

Issue/Introduction

The certificates on TAS foundation are rotated successfully. However, warning is still shown on Grafana for some certificate although the new expiration date of the certificate is several years away as illustrated by the below images. 

Certificate verification error was also seen in opsman-cert-expiration-exporter job logs on cert-expiration-exporter instance of healthwatch2-pas-exporter deployment. For example,

{"timestamp":"2025-03-06T06:16:46.624951378Z","level":"error","source":"exporter","message":"exporter.error fetching expiring certificates from opsman","data":{"error":"Failed to get BOSH trusted certificates with error: failed to get staged director properties: could not send api request to GET /api/v0/staged/director/properties?redact=true: Get \"https://pcf.example.com:443/api/v0/staged/director/properties?redact=true\": Post \"https://pcf.example.com:443/uaa/oauth/token\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}}

 

Environment

• Tanzu Platform for Cloud Foundry
• Healthwatch for VMware Tanzu

Cause

The opsman-cert-expiration-exporter job will periodically call Ops Manger API to get latest state of all certificates. Since a secure communication over TLS is required for making Ops Manager API call, a proper CA certificate is needed on the BOSH instance cert-expiration-exporter for validating TLS certificate returned by Ops Manager. 

The "tls: failed to verify certificate: x509: certificate signed by unknown authority" error indicates the Ops Manager certificate couldn't be verified on BOSH instance cert-expiration-exporter.

Resolution

User can configure custom certificate for Ops Manager or use default self-signed certificate. More details are available in the Ops Manager document. 

Using self-signed certificate

 

On BOSH Director Settings page (Security pane): 

Enable Include Tanzu Ops Manager Root CA in Trusted Certs  checkbox.

OR

Put Ops Manager root CA in the Trusted Certificates box.  

Ops Manager root CA can be downloaded on the UI (Settings -> Advanced Options)

NOTE: "Apply Changes" is required against the "Healthwatch2 PAS Exporter" tile.

 

Using custom certificate

Put the root CA of custom certificate in the Trusted Certificates box on BOSH Director Settings page (Security pane). 

NOTE: "Apply Changes" is required against the "Healthwatch2 PAS Exporter" tile.

OR

Put a chain of certificates, which include the root CA for the custom certificate, in Ops Manager Settings page ("SSL Certificate" pane).