Monitoring metric data quality

Products

Insights Observability Wavefront by VMware Aria Operations for Applications

Issue/Introduction

Quick bursts of high cardinality metrics can cause performance issues when querying data. Monitor and investigate metrics that have higher cardinality will avoid "NO DATA" from being returned on Dashboard and Alerts.

Cause

During ingestion, Wavefront assigns an ID to each newly added metric name, span name, source name. Storing high-cardinality data like timestamps, web session ids, login ids etc. in a metric name, source name or point tags creates more district values requiring a unique IDs for each.

Resolution

Tanzu Observability Usage

Monitor Tanzu Observability usage

The Usage integration includes predefined alerts to monitor new metrics being sent, which may include metrics with potentially problematic data quality

To install the predefined Wavefront usage alerts, click the Install All button under the Wavefront usage integration tab.
After the alerts are installed, an alert target needs to be assigned to receive notification of the alert being triggered.
In order to make any changes to the predefined alerts, they need to be cloned first.

Information on each of the Alerts listed above

High rate of host IDs observed:

Predefined alert that is designed to keep track of any bursts of new source IDs. The newly created source IDs may be due to an expected flow in the metrics pipeline, (new containers spinning up etc.), or due to a configuration issue that needs investigation.
Alert for tenant is reporting high rate of new host (source) IDs to Tanzu Observability.

High rate of metric IDs observed:

Predefined alert that is designed to keep track of new metric IDs sent in per minute over a 10-minute moving average. See example of data shape below that would trigger this alert.
The metrics shown, have a session id included in the metric name; this doesn't add any value as it is essentially in the format of an event instead of a continuous time series.
The querying of this type of metric data would require a wildcard (*) and could experience performance problems due to the poor-quality data shape.

"http.client.requests.clientName.api-test.<server>.method.GET.outcome.SUCCESS.status.200.uri.-v2-order_carts-xxxxx-xxxx-xxxx-xxxx-delivery_times-.count_95 2669831779"
"http.client.requests.clientName.api-test.<server>.method.GET.outcome.SUCCESS.status.200.uri.-v2-order_carts-xxxxx-xxxx-xxxx-xxxx-.upper 2669831781"

High rate of string IDs observed:

Predefined alert that is designed to keep track of new string IDs or point tags.
A high rate of new point tags added to a metric, if left unchecked, can have a detrimental effect on the performance of the queries associated with the metric.
See example below, which has a string of characters in the point tag, which would quickly ramp up cardinality.

Point Tag: com.docker.swarm.task.name=xxx-665-amp.1.xxxxxxxxxx 1610985074
Point Tag: com.docker.swarm.task.name=xxx-665-amp.1.xxxxxxxxxx 1610985074

Spying new IDs Use Wavefront SPY - See Section: Get New ID Assignments with Spy for additional URL suggestions.

The Wavefront instance includes an HTTP endpoint that can be leveraged to provide a window into the current stream of new IDs. Using this endpoint can quickly expose problematic new ID data points.

To get a list of new ID assignments, open a browser and use the following endpoint URL, replacing <cluster> with the name of the Wavefront cluster name.

https://<cluster>. wavefront.com/api/spy/ids

The Api Spy will sample the new ID assignments and print the metrics associated with them out on screen.

Using WFtop to spy new IDs.

WFtop is a utility that downloads and installed locally to a machine. This program will also allow for ID exploration. Please see the Wftop documentation for pre-requisites, steps and details of installation.

Wftop can also be used to spy on the new IDs being ingested. Once connected to the instance with wftop, change the configuration to spy on 'Id Creations' with a sampling rate of 1.0 (100%) and then select one of the metric, host, histogram or span types and the new IDs will be displayed in the Wftop pane.