Unexpected error in flush loop java.lang.OutOfMemoryError: Java heap space

Products

• Aria Operations for Applications
• Wavefront by VMware
• DX OpenExplore
• Observability

Issue/Introduction

Error sending data to Wavefront: java.lang.IllegalStateException: Connection pool shut down.  Proxy does not resurrect and requires manual intervention.

Cause

Too much data is being sent or scrapped by to the Wavefront Proxy. The proxy does not have the resources to keep up with the ingestion levels.

Resolution

Upgrade to Wavefront Proxy 13.7 [GA Release], which includes an Updated Netty netty-bom to version 4.1.108.Final to resolve CVE-2024-29025. 

Once upgraded, proxy crashes may continue for other reasons but now will resurrect the pod automatically. 

Additional steps to resolve other OOM issues AFTER upgrading the proxy begin with understand the data patterns and data needs of each cluster environment.

• Review out-of-the-box Dashboard: Tanzu Observability Service and Proxy Data for information on ingestions rates, Queuing Reasons, P99 Data Received Lag, ect.
• Review charts on your Kubernetes Clusters for metrics including namespaces that are not necessary to ingest and can be filtered out.

Once an understanding of your data patterns and needs you can:

• Reduce the amount of ingesting flowing through the proxy by enabling Operator Metrics filtering and/or Advance Pre-Processor Rules on the proxy container.
• Increase the CPU Limits on the containers
• Increase the Memory Resources on the containers.
• Deploy multiple Proxys and distribute ingestion load across all.  You can deploy a Load Balancer in front of your proxies.
• For non-production | Test | Stage environments, update the defaultCollectionInterval on the collectors from 60s to 300s. This will result in scraping the data ever 5 minuets instead of every 1 minuet.  Reducing ingestion by 1/5th.

Additional Information

wavefrontHQ/observability-for-kubernetes - wavefront-full-config.yaml

wavefrontHQ / Observability for Kubernetes - The Kubernetes Metrics Collector supports filtering metrics and events.

Defining proxy pre-processor rules - Advanced-proxy-preprocessor-rules

Install and Manage Wavefront Proxies - See Section: Proxy Host Requirements

Wavefront Proxy - Section: Production Environment: Team of Proxies Load Balancer

Monitor Wavefront Proxies - Section: Examine the Health and Usage of a Proxy with the Proxy Dashboard