HealthWatch Grafana VM health dashboard shows gaps intermittently

Products

• VMware Tanzu Application Service

Issue/Introduction

On Healthwatch Grafana dashboard, all VMs health state except for BOSH director are reported with a gap, though no instances or any jobs inside those instances are unhealthy. 

 

Environment

• TAS 2.0 and above
• Healthwatch v2.x

Cause

The issue usually occurs on large scale platform. 

VM health is one type of gauge metrics. The component pas-exporter-gauge in Healthwatch Exporter for TAS is in charge of transferring gauge data from TAS firehose v2 to Healthwatch prometheus. The gauge metrics come from 4 major sources:

• system metrics(such as CPU, memory, disk usage) from all instances 
• TAS job metrics 
• app container metrics on TAS
• job metrics from other tile such as RabbitMQ

Currently one single pas-exporter-gauge can not process the data stream fast enough so that some metrics get dropped by doppler and reverse-log-proxy(firehose v2). This can be confirmed with CF CLI log-cache-plugin, with steps:

1. bosh ssh  into Healthwatch Exporter deployment, pas-exporter-gauge instance 
2. netstat -anpt | grep 8082 to identify which TAS trafficontroller instance the pas-exporter-gauge process connects to. 
3. cf tail reverse-log-proxy -f --json | egrep "counter.*dropped" | grep <IP_ADDRESS>

In the case the output show delta >0 and increasing total dropped value, it could prove that the single pas-exporter-gauge can not consume gauge metrics fast enough. 

{"timestamp":"1723131798369992000","source_id":"reverse_log_proxy","instance_id":"","deprecated_tags":{},"tags":{"deployment":"cf-54926cb29f45e0241f81","direction":"egress","index":"522be8e9-eb4e-44be-9c50-f7f716cc40d0","ip":"10.225.30.216","job":"loggregator_trafficcontroller","metric_version":"2.0","origin":"loggregator.rlp","product":"VMware Tanzu Application Service","system_domain":"###.###.###"},"counter":{"name":"dropped","delta":"2343","total":"34694540"}}

Resolution

As pas-exporter-gauge can not be scaled to multiple instances, you can try to either or all below workaround: 

• vertically scale pas-exporter-gaute with more CPU
• increase TAS > System Logging > System metrics scrape interval from 15s to bigger value like 60s
• increase RabbitMQ(if it is on the platform) metrics polling interval from 30s to bigger value like 60s

those workaround will add more CPU resource and reduce data volume.