Error connecting to the WebSocket (WSS) through F5 Load Balancer

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

This issue manifests itself when trying to connect to a Loggregator WSS endpoint and the connection fails. You can try accessing the WSS endpoint in many different ways including via JMX Bridge, cf logs, or when running an errand. You may also experience the OpenTSDB Firehose nozzle VM appearing to go up and down in JMX Bridge Tiles Status tab.   

Errors: JMX Bridg or OpenTSDB Firehose nozzle

You may receive error messages such as the following:
 

2016/08/01 13:02:05 Error while reading from the firehose: websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 Disconnected because nozzle couldn't keep up. Please try scaling up the nozzle.
2016/08/01 16:02:05 Closing connection with traffic controller due to websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 OpenTSDB Firehose Nozzle shutting down...
2016/08/01 16:02:05 Starting OpenTSDB Firehose Nozzle...
2016/08/01 16:02:05 Error while reading from the firehose: websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 Disconnected because nozzle couldn't keep up. Please try scaling up the nozzle.
2016/08/01 16:02:05 Closing connection with traffic controller due to websocket: close 1008 Client did not respond to ping before keep-alive timeout expired.
2016/08/01 16:02:05 OpenTSDB Firehose Nozzle shutting down...
2016/08/01 16:02:06 Starting OpenTSDB Firehose Nozzle...

 

Error: cf logs

The following error is received when trying to connect via the command line using cf logs.

cf logs <App_Name>

 websocket: close 1008 (policy violation): Client did not respond to ping before keep-alive timeout expired.          

Error: Errand

The following error is received when running errands.

websocket: close 1008 (policy violation): Client did not respond to ping before keep-alive timeout expired.
panic: sync: negative WaitGroup counter
goroutine 412 [running]

Environment

Cause

The Loggregator API endpoint must be accessible via a WebSocket (WSS). This problem is caused by an issue with the firmware on your F5/BIGIP dropping WSS packets. This issue applies to versions of F5 Firmware prior to version 11.4.0. Please check if this issue is applicable to the version of Firmware you are currently running on your F5.

Resolution

If the issues and versions of the software outlined above are applicable to you, please apply the following:

Workaround

1. Use an iRule to selectively disable the HTTP profile for WebSocket traffic, as outlined here [2].

2. https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

Note: In the link above several solutions are given, the iRule change is confirmed to work with PCF. 

3. F5 /BIG-IP support for the WebSocket protocol [3] is available here.