Troubleshooting missing metric data in DX OpenExplore formally Wavefront.

Products

• Wavefront by VMware
• Aria Operations for Applications
• Insights

Issue/Introduction

Symptoms of missing data in DX OpenExplore(OE) include:

• Reviewing chart for a specific time range shows empty or missing data.
• Chart showing broken lines.
• Dashboard, Charts, Alerts display "No Data".
• Pushing data to OE via wavefront proxy, we are seeing missing data points.

Resolution

Obsolete Data

If a time series hasn’t ingested any data points for a certain period of time (by default 2 weeks) we consider the time series obsolete.  By default, obsolete data is not included in charts to improve chart loading. If you want to see the data:

• In the chart, click Advanced and select Include Obsolete Metrics.
• In the dashboard, you can turn on obsolete metrics for each chart, but performance can suffer.

Hidden Metrics

To view hidden metrics:

1. Select Browse > Metrics.
2. Click the Manage Hidden Metrics button.
3. Click the Unhide button to the right of the metric or metric prefix to unhide
4. Click Save.

Wavefront Data Format Syntax

Verify the metric adheres to the acceptable format syntax.

<metricName> <metricValue> [<timestamp>] source=<source> [pointTags]

• Each field must be spaced-separated.
• Each line must be terminated with the newline character (\n or ASCII hex 0A)
• For more details see Data Format - Metrics Data Format Syntax

Preprocessor Rules

• Check for configuration errors in the proxy preprocessor rules
• Check for unintended restrictive rules in the proxy preprocessor rules.
• Verify code to block metrics
• Verify code to replace characters or strings.
• Check regex filters included in the YAML file are correct. If necessary, test with an online regex editor of your choice to confirm accuracy. 

Metrics (points) Log as blocked points at proxy.

• Points may be blocked at the proxy level, to track points being blocked see:
• Proxy Configuration - Blocked Data Log
  • Enable blocked point logging.
  • On the proxy, typically located here:  /etc/wavefront/wavefront-proxy/log4j2.xml.
  • Edit the log4j2 configuration fileUncomment the blocked points file appender as per the example below:
    
    <!--
        <AppenderRef ref="BlockedPointsFile"/>
    -->
• Kubernetes Troubleshooting

Aged Data Deletion Policy

• Check with Support regarding your current metric retention policy to verify the data is not older than retention policies. 
• Metrics, Distribution, Traces all have different retention policies. Generally Metrics are retained for 400 days while Distributions (Histograms) are retained or 7 days. 
  • Note: Currently Traces are not being ingested under DX OpenExplore however they are on the roadmap for future inclusion.

Ephemeral Metrics

• Converting a persistent metric to ephemeral permanently deletes the data points of that metric that are older than 28 days.
• For more information on this please see "Change the Retention Period of Metrics" section of TechDocs:  Metrics and the Metrics Browser

Additional Information

FAQ - How Is Data Retained and Aggregated In DX OpenExplore?

If reason for missing data is still unknown open a ticket with the Support Team.

Include the below for faster resolutions

• Proxy Name 
• Proxy Logs
• Proxy Preprocessor Rules
• Integration Type and Account Identifier 
• Integration Allow or Block configuration.