Unexpected error in flush loop java.lang.OutOfMemoryError: Java heap space

Products

• Aria Operations for Applications
• Wavefront by VMware
• DX OpenExplore
• Observability

Issue/Introduction

Proxy crashes and does not resurrect and requires manual intervention.

Proxy logs contain: Error sending data to Wavefront: java.lang.IllegalStateException: Connection pool shut down.  

Cause

Proxy does not have enough resources to keep up with amount of data that is being processed through it.

Resolution

Phase 1: Immediate Remediation (Upgrade)

Upgrade to Wavefront Proxy 13.7 [GA Release], which includes an Updated Netty netty-bom to fix version 4.1.108.

• Wavefront Proxy 13.7 [GA Release]
• CVE-2024-29025

Phase 2: Long-term Optimization (Data Analysis)

It is recommended to review your proxies ingestion to understand the data patterns and ingestion needs. 

• Review out-of-the-box Dashboard: Tanzu Observability Service and Proxy Data for information on ingestions rates, Queuing Reasons, P99 Data Received Lag, etc.
• Review charts on your Kubernetes Clusters for metrics including namespaces that are not necessary to ingest and can be filtered out.

Once an understanding of your data patterns is achieved, below are recommendation you can implement to better utilize the proxies in your enviroment.

• Reduce the amount of ingesting flowing through the proxy by enabling Operator Metrics filtering and/or Advance Preprocessor Rules on the proxy container.
• For non-production | Test | Stage environments determine if data is needed every 1-minute(60 seconds) or can you decrease that to every 5-minute(300 seconds)? 
• Increase the CPU Limits on the containers
• Increase the Memory Resources on the containers.
• Deploy a load balancer and distribute ingestion load across multiple Proxies

Additional Information

• wavefrontHQ/observability-for-kubernetes - wavefront-full-config.yaml
• wavefrontHQ / Observability for Kubernetes - The Kubernetes Metrics Collector supports filtering metrics and events.
• Defining proxy pre-processor rules - Advanced-proxy-preprocessor-rules
• Install and Manage Wavefront Proxies - See Section: Proxy Host Requirements
• Wavefront Proxy - Section: Production Environment: Team of Proxies Load Balancer
• Monitor Wavefront Proxies - Section: Examine the Health and Usage of a Proxy with the Proxy Dashboard