How to Troubleshoot Proxy Preprocessor Rules issues

Products

• Insights
• Observability
• Wavefront by VMware
• Aria Operations for Applications
• DX OpenExplore

Issue/Introduction

General troubleshooting steps for resolving issues related to preprocessor rules when using Wavefront Proxy, including in Kubernetes environments. If you are experiencing problems with your preprocessor rules not taking effect or not being applied correctly, the following steps will help you diagnose and resolve the issue.

Resolution

The Wavefront proxy offers a preprocessor feature that applies rules to data before sending it to Operations for Applications. These rules are useful for resolving data quality issues that cannot be fixed at the source or block undesired data.

Procedure

Before you begin, take a moment to review this documentation: Configuring Wavefront Proxy Preprocessor Rules, and ensure you have clear understanding of the information before proceeding.

1. What type of data is the target to apply the rule (timeseries, spans, logs)?
2. Is it just one rule that is causing the issue or concern?  Or Multiple
   1. Are there other rules working properly?
   2. Compared format and syntax.
3. Rules are applying to the correct ports and data type (span, timeseries, logs) See Section:  Applying Rules to Multiple Ports from the above article.

 

1. Assess the effectiveness of the configured rules.

A good start is to ensure that the preprocessor rules itself are working as expected. Some useful charts are:

• Access the Proxy in question, found under Browse menu, Proxies. 
  • Overview Tab: Go to Section - Metrics, view Chart - Blocked Points Per Second
  • Chart should have activity and not flat 0 (zero) which could indicate Proxy is not processing any rules.
    
    
  • Proxy Configuration Tab: Go to Card - General (starting proxy 12.4 only)
  • Make sure preprocessorConfigFile line is populated with file details.
    • If empty review the wavefront,conf file for missing or commented out preprocessorConfigFile.
      
      
• Review Tanzu Observability Service and Proxy Data Dashboard. 
  • Go to Section Proxy Troubleshooting.
  • Chart Preprocessor Rules: Hit Ratio to identify if there is a specific preprocessing rule not hitting any points = flat 0 (zero) line.
    
    

2. Check Regex effectiveness.

The Wavefront proxy uses Java-style regex pattern, when creating preprocessing rules, it is important to understand the impact of having an accurate regex pattern and matches all scenarios. 

Please review the Regex Notes to ensure the expression follows best practices.  You can leverage the use of third-party websites such as regex101 or ChatGPT to build complex regex patterns and validate customer regex.

For example, from below screenshot proxy will only consider <prodDC1> string as match, since other strings are not full match. 

 

4. Consider Enabling Blocked Points.

Enabling the block point feature could be helpful in troubleshooting preprocessor rule issues. 

Refer to Proxy Troubleshooting See Step 2: Enable Blocked Point Logging and Examine Blocked Points.

Open the <wavefront_config_path>/log4j2.xml configuration file and uncomment the BlockedPointsFile section and RawBlockPoints Logger and set the level="INFO".

Final result should look like this: BlockedPointsFile section

RawBlockPoints Logger

Examine the blocked points entries in the <wavefront_log_path>/wavefront-blocked-points.log file.

Once testing is done, make sure to disable blocked logging as this can produce massive data.

Note: Modifying log4j2.xml file under a containerized Wavefront Proxy pods is not possible.

 

5. Analyze Raw Metrics with Spy.

To gain deeper insights into your metrics, you can leverage the Spy feature. By accessing the Spy API endpoint, you can retrieve raw metric data and conduct further investigation with advanced filtering capabilities. To access the Spy API, use the following URL format:

https://<yourClusterName>.wavefront.com/api/spy/points?proxy:<yourProxyName>

This endpoint allows you to capture a sample of metric data directly and enables you to apply various filters to narrow down your analysis.

For example, you can use it to identify timeseries that are still showing a certain metric despite the fact that it is not actually coming through the specified proxy. It can also help you detect cases where a preprocessor rule is functioning properly, but certain data points not related are slipping past your charts. For detailed instructions on how to effectively use the Spy, please refer to:

• Use Wavefront Top or Spy to Investigate Traffic
• VMware Aria Operations for Applications Spy to Investigate Metrics per Proxy

 

6. Additional validation steps for Kubernetes environments.

Preprocessor rules in Kubernetes are seamlessly transmitted to the Wavefront proxy by leveraging the ConfigMap Kubernetes object, which serves as a reliable repository for storing essential configuration data, conveniently accessible by containers executing within a pod.

Note: values enclosed by <> are those that need to be changed.

1. When applying incorrect rules, pod will be in a CrashLoopBackOff state with multiple restarts, check Proxy pod is in running state, by running

kubectl get pods -n <IntegrationNamespace>

2. You can get logs by running:

kubectl logs <wavefront-proxy-pod> -n <IntegrationNamespace>

You could see syntax errors showing, also the startup messages showing, pod trying to mount preprocessor rules ConfigMap:

If you don't see relevant errors, make sure Preprocessor rules are being successfully mounted to the pod, you can run below command to get the volume mounts in the proxy pod:

kubectl get pod <wavefront-proxy-pod> -n <IntegrationNamespace> -o custom-columns=VOLUME_MOUNTS:.spec.containers[*].volumeMounts[*].name,MOUNT_PATHS:.spec.containers[*].volumeMounts[*].mountPath

If you don't see the mount for the preprocessor rules, probably the preprocessor rules are not being correctly deployed, make sure official documentation is followed.

3. You can use the following command to inspect inside the pod:

kubectl exec -it <wavefront-proxy-pod> -n <integrationNamespace> -- /bin/bash

From the above step you can get the mount path, in this case /etc/wavefront/preprocessor/rules.yaml using this command:

cat /etc/wavefront/preprocessor/rules.yaml

Ensure that the output of the last command displays the expected preprocessor rules.

If you have made changes to the ConfigMap it's important to note in some scenarios, pod will not automatically reflect these changes. In order to apply the new ConfigMap map configuration, you need to restart the pod by deleting it. 

kubectl delete

Note: For Production environments make sure pod does not have backlog, if you delete it, this backlog will be lost, and just rely on:

<kubectl apply -f config.yaml >