"One or more metrics and their origins were not found!" error when running Healthwath errand "validate-expected-metrics"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When running the errand validate-expected-metrics through Operations Manager (Ops Manager) or BOSH, it fails with the error:

FAILURE! One or more metrics and their origins were not found!  
 [ "mysql./mysql/available", "mysql./mysql/galera/wsrep_ready", "mysql./mysql/galera/wsrep_cluster_status", "mysql./mysql/galera/wsrep_cluster_size" ] 

This errand is checking the Healthwatch MySQL database for certain Tanzu Application Service (TAS) MySQL VM metrics, but they are not found.

You can confirm this by following these steps:

1. Run this command: bosh -d p-healthwatch-DEPLOYMENT ssh mysql/0

2. Run the command: sudo -i

3. Run the command: mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf

4. Use: platform_monitoring;

5. Run these queries:

select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/available';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_ready';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_cluster_status';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_cluster_size';

All queries return a 0 count, meaning the errand is not finding the expected data.

The primary reason for this issue is your TAS platform is configured to use an external MySQL database, not an internal MySQL database. You can confirm this in the TAS tile > Databases.

Resolution

The internal MySQL VM runs a job called 'cluster-health-logger' and the logs can be found in /var/vcap/sys/log/cluster-health-logger/cluster_health.log. Healthwatch is not getting these logs when TAS is configured to use an external database and the errand fails.

You do not need to enable this errand during Healthwatch deployment. It it is off by default and this can be seen in the tile under Errands > Metrics Troubleshooting.

Note: If you would like to use it for troubleshooting, it only supports deployments using an internal MySQL database.

If you are using an internal database and still see those errors, follow the steps below to ensure MySQL metrics are being populated into the Healthwatch database.

1. Run this command: bosh -d p-healthwatch-DEPLOYMENT ssh mysql/0

2. Run the command: sudo -i

3. Run the command: mysql --defaults-file=/var/vcap/jobs/pxc-mysql/config/mylogin.cnf

4. Use: platform_monitoring;

5. Run these queries:

select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/available';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_ready';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_cluster_status';
select count(*) from value_metric_agg where origin='mysql' and name = '/mysql/galera/wsrep_cluster_size';