DXOE Proxy shows a partial drop with ingestion

Products

• DX OpenExplore

Issue/Introduction

We are seeing a drop in ingestion and it never backfills.

Customers may observe a dip in metrics ingestion and no pending pushback or queuing, including internal Pub/Sub queues.

This typically occurs when the Wavefront proxy is not receiving data from its designated sources, even if the backend services are healthy.

Environment

• DX OpenExplore (SaaS)
• Wavefront Proxy (All Versions)

Cause

The Wavefront proxy is not receiving data from the data source. This is usually caused by network-related issues, such as firewall rules, routing changes, or the data source service failing to transmit to the proxy's ingestion port.
In this example there was no pushback or queuing of the proxy, including internal Pub/Sub queues

.
No issues where found on the DXOE side at that time.

Resolution

1. Verify if metrics are reaching the proxy by checking the proxy logs (wavefront.log).
2. Search for "no data received" or connection timeout errors in the logs of the service emitting the metrics.
3. Confirm there is no backend pushback by reviewing the Observability Service and Proxy Data Dashboard.
4. Test connectivity between the data source and the proxy using tools like telnet or curl on the ingestion port (default 2878).

Additional Information

• Proxy Troubleshooting Guide
• Contact Broadcom Support