"Remote Error, tls: handshake" failures from Loggregator when using an F5 Load Balancer

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

The purpose of this article is to help you get past a known issue with the Loggregator in Elastic Runtime 1.10.x. If you are using HA_Proxy, you will likely not run into this issue. This issue has been observed in Pivotal Cloud Foundry® environments that are using an F5 load balancer and were upgraded to 1.10.x.

 

Environment

Cause

Once your PCF environment, utilizing an F5 Load Balancer has been upgraded to Pivotal Cloud Foundry® 1.10.x, you will start to see the errors similar to the following generated from the Loggregator VM:

2017/05/16 01:59:36 Error getting auth data: Post https://uaa.<sys-domain>/check_token: remote error: tls: handshake failure 
2017/05/16 01:59:41 Could not get app information: [Get https://api.<sys-domain>/internal/log_access/b4d55fd2-557e-46ea-8054-fb70d722d7da: remote error: tls: handshake failure]

If you are running an Elastic Runtime version prior to 1.10.6 you may also see a similar error coming from your GoRouter VM:

[2017-04-12 14:42:35+0000] 2017/04/12 14:42:35 http: TLS handshake error from 192.168.254.225:42020: tls: no cipher suite supported by both client and server 

Resolution

In order to resolve this issue, you will need access to your F5 Load Balancer Configuration, Ops Manager Dashboard for your PCF 1.10 environment, and the ability to execute Bosh commands against your PCF 1.10 Environment.

Step 1

In your Ops Manager Dashboard, Click on the Elastic Runtime Tile -> Settings Tab -> Network Section -> and make sure you have the option check to Forward unencrypted traffic to Elastic Runtime Router. See below:

After doing this, you'll likely need to click on "Apply Changes" if this was not the setting you had already selected. Otherwise, you can simply restart your  Loggregator  VM following Step 2.

Step 2

As of Elastic Runtime 1.10.9, the Loggregator only supports the following two Ciphers Natively:

var supportedCipherSuites = []uint16{
    tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
    tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
}

You need to make sure that the F5 Load Balancer will support those two Ciphers, in lieu of your current cipher suite in the F5 settings. The goal is to add ECDHE+AES-GCM in the supported cipher suites. Otherwise, F5 will not be able to negotiate a cipher with the Loggregator HTTP client, and F5 will reject the request resulting in TLS handshake failure.

To add to the supported cipher suites on F5, you'll need to add the following profiles in the supported cipher suite. The following example adds support for both the default ciphers and the two GCM ciphers supported by the Loggregator:

DEFAULT:ECDHE+AES-GCM

Once you have added that profile in your supported cipher suites, restart the Loggregator VM. You should do this by running the following command while on the Loggregator VM:

monit restart loggregator_trafficcontroller

Once you have restarted the Loggregator, you shouldn't see the tls: handshake failures anymore. You can verify the system is talking again by running the Elastic Runtime smoke-tests errand.