Error sending points to Wavefront (HTTP -1). Data will be requeued and resent

Products

• Insights

Issue/Introduction

An application receives the below error from the VMware Aria Operations for Applications proxy.

11:15:49.359 [wavefrontClientSender-0] WARN c.w.s.common.clients.WavefrontClient - Error sending points to Wavefront (HTTP -1). Data will be requeued and resent.
11:15:49.360 [wavefrontClientSender-0] ERROR c.w.s.c.c.service.ReportingService - Unable to obtain status code from the Wavefront service at http://wavefront-proxy:2878/report?f=wavefront
java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
at java.base/sun.net.www.http.HttpClient.New(Unknown Source)

Note: The preceding log excerpts/Messages are only examples. Date, time, and environmental variables may vary depending on your environment.

Error Description: There could be network connectivity issues between the emitting service and the proxy or something else caused java.net.SocketTimeoutException: connect timed out

Environment

• Data Ingestion
• Product edition: All versions
• Feature Category: Application Integration

Resolution

The error is safe to ignore if you're not seeing other exceptions, or a log at WARN level with a message containing "Buffer full, dropping.." The reason why you see this as the data is queued and retried later, at which time it sounds like WavefrontClient doesn't get a timeout when connecting to the WF proxy.

Note: To check if metrics are being dropped in your application by WavefrontClient, you can query for ~sdk.java.core.sender.wfclient.points.dropped, with a source of your application's hostname (or some other uniquely identifying tag).

But if you are seeing buffer full, dropping, warning message and exception repeatedly follow the below steps:

• Try increasing the batch size of your WavefrontClient, which is recommended in the error log about "Buffer full, dropping..."
  The default batch size is 10,000. Try increasing the batch size and see if the issue persists. Instructions for increasing the batch size in the VMware Aria Operations for Applications Java SDK are mentioned at Initialize the WavefrontClient.
• You can also consider increasing the maximum in-memory queue size to something larger than already set (note: the document linked above incorrectly states the default queue size is 50,000). This will give more space/time before the queue fills up and points are dropped.