Connection resets from metrics.wavefront.com

Products

• DX OpenExplore

Issue/Introduction

Customers using legacy versions of the Wavefront Proxy (specifically version 11.3 and older) may experience massive data backlogs and recurring connection errors when the tenant reaches its ingestion limit.

Symptoms

• Error Messages: Connection reset, Broken pipe (Write failed), or HTTP 429 errors in proxy logs.
• Backlog Growth: Points become backlogged in the proxy spool (e.g., millions of points or GBs of disk usage) and fail to drain even after traffic returns to normal levels.
• Selective Failure: Small manual POST requests (e.g., via curl) may succeed, while the proxy's large batch flushes (100K points or ~5-10MB payloads) consistently fail.

Cause

This issue is typically caused by a combination of factors:

1. Tenant-Level Ingestion Limits: DX OpenExplore enforces limits at the tenant level. If one proxy or a group of proxies exceeds the contractual Points Per Second (PPS) limit, the backend issues HTTP 429 (Too Many Requests) responses.
2. Legacy Proxy Defect: Version 11.3 and older contain a known issue where retrying large failed batches results in connection resets and broken pipes rather than graceful back-off. This prevents the backlog from ever draining effectively.
3. Shared Quota: Since limits are tenant-wide, a burst from one proxy can trigger pushback for all other proxies sharing the same token.

Resolution

### Step 1: Upgrade the Proxy
Upgrade all affected proxies to **Version 15** or later. Newer versions include critical stability improvements for connection pooling and retry logic that resolve the `Connection reset` loop.

### Step 2: Configure pushRateLimit
To prevent a single proxy from consuming the entire tenant budget and triggering a system-wide pushback, configure a local rate limit on each proxy.
- Edit the proxy configuration file.
- Add or update the following parameter:
  `pushRateLimit=<limit_value>`
- Set the `<limit_value>` based on the specific proxy's expected throughput relative to your total tenant capacity.

### Step 3: Clear Stale Backlogs
After upgrading, monitor the spool directory (usually `/var/spool/wavefront-proxy/`). If the backlog remains stagnant:
1. Stop the proxy service.
2. Verify the spool file sizes.
3. Restart the proxy service to trigger a fresh drain of the queued data.