"[Fail] Bosh metric ingestion [It] Ingests metrics from the director into MySQL" error when Healthwatch smoke test fails in Operations Manager 2.10.7

Products

• Operations Manager

Issue/Introduction

When using Operations Manager (Ops Manager) 2.10.7, the Healthwatch smoke test fails with the following error:

 Failure [120.001 seconds]
Bosh metric ingestion
/var/vcap/data/compile/healthwatch-data/src/github.com/pivotal-cf/healthwatch-data/data-ingestion/smoketests/bosh_metrics_test.go:14
Ingests metrics from the director into MySQL [It]
/var/vcap/data/compile/healthwatch-data/src/github.com/pivotal-cf/healthwatch-data/data-ingestion/smoketests/bosh_metrics_test.go:45

Timed out after 120.000s.
Bosh Metrics should be stored in MySQL from the Firehose. Check logs for the healthwatch-ingestor and healthwatch-worker applications. Also verify BOSH system metrics are flowing through the Firehose.
Expected
<bool>: false
to be true

/var/vcap/data/compile/healthwatch-data/src/github.com/pivotal-cf/healthwatch-data/data-ingestion/smoketests/bosh_metrics_test.go:50
------------------------------


Summarizing 1 Failure:

[Fail] Bosh metric ingestion [It] Ingests metrics from the director into MySQL
/var/vcap/data/compile/healthwatch-data/src/github.com/pivotal-cf/healthwatch-data/data-ingestion/smoketests/bosh_metrics_test.go:50

Ran 5 of 5 Specs in 140.445 seconds
FAIL! -- 4 Passed | 1 Failed | 0 Pending | 0 Skipped --- FAIL: TestSmoketests (140.44s)
FAIL

Environment

Product Version: 2.10

Resolution

This issue has been resolved in Ops Manager 2.10.8.

This is an issue specific to Ops Manager 2.10.7. On the BOSH Director VM, checking the log file /var/vcap/sys/log/health_monitor/health_monitor.stderr.log will show an error similar to the one below:

/var/vcap/jobs/system-metrics-server/bin/bosh-monitor/run.sh: line 5: /var/vcap/sys/log/system-metrics-server/system-metrics-plugin.log: No such file or directory

If you check the directory it is referring to, /var/vcap/sys/log/system-metrics-server/, you should see that the file system-metrics-plugin.log file is missing. This prevents the system metrics server from starting up on the BOSH Director correctly, which in turn causes the healthwatch smoke test to fail.

Workaround

As a workaround, the script for the metrics server can be edited. This is located at /var/vcap/jobs/system-metrics-server/bin/bosh-monitor/run.sh on the BOSH Director.

You can edit this script as follows:

• Edit this line:

exec $PACKAGE_PATH/system-metrics-plugin --server-port="25594" &>> ${LOG_DIR}/system-metrics-plugin.log

to:

exec $PACKAGE_PATH/system-metrics-plugin --server-port="25594" # &>> ${LOG_DIR}/system-metrics-plugin.log

Doing a monit restart system-metrics-server should then bring up the process correctly. This should allow the smoke test to pass, as long as the BOSH Director is not recreated.