Healthwatch TAS monitoring shows metric gaps
Article ID: 298321
Updated On:
Products
VMware Tanzu Application Service for VMs
Issue/Introduction
In large TAS environments (300+ VM's), we have seen that Grafana can show some gaps in the charts as below.
Environment
Product Version: 2.11
Resolution
System-metric-scraper has an one-second timeout when scrapes remote system-metric-agent endpoint which is insufficient when handling large number of VMs.
To resolve the issue, the timeout has been adjusted to one minute in system-metrics-scraper release v4.0.2.
TAS 2.11.47+, 2.13.29+, 4.0.10+, 5.0.1+ includes v4.0.2+ that solves this issue.
The timeout is not configurable on TAS UI, so in case you can't update at the moment or want to test the change, the available workaround is to edit /var/vcap/jobs/loggr-system-metric-scraper/config/bpm.yml on clock_global VM and add `SCRAPE_TIMEOUT: 15s` under `SCRAPE_INTERVAL: 15s` and run 'monit restart loggr-system-metric-scraper'
To resolve the issue, the timeout has been adjusted to one minute in system-metrics-scraper release v4.0.2.
TAS 2.11.47+, 2.13.29+, 4.0.10+, 5.0.1+ includes v4.0.2+ that solves this issue.
The timeout is not configurable on TAS UI, so in case you can't update at the moment or want to test the change, the available workaround is to edit /var/vcap/jobs/loggr-system-metric-scraper/config/bpm.yml on clock_global VM and add `SCRAPE_TIMEOUT: 15s` under `SCRAPE_INTERVAL: 15s` and run 'monit restart loggr-system-metric-scraper'
Feedback
Yes
No
Powered by
