Route services are timing out with 502s with transparent Load Balancers

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Transparent TCP (Layer 4) Load Balancers such as Azure's Load Balancer or AWS Network Load Balancer do not support TCP connection where source and destination IP are the same.


Gorouter supports the concept of Route services. This provides a way to filter or intercept incoming requests. Among many things, a router service can be used to limit the amount of requests to the app. See the docs here for more details on route services.

Resolution

Route services are implemented in Gorouter and will cause the initial request to be forwarded to your custom route service app, which can then apply logic that dictates the fate of the request. It can drop the request or allow it through to the actual app. Unfortunately, if the Gorouter has to deliver the request to a route service and the route service is an app on the foundation, this causes Gorouter to send a request to your load balancer and back into Gorouter. As mentioned above, this cause both the source and destination addresses to be Gorouter which is not supported on some load balancers. In practice, this will result in requests that timeout with 502 and have no response.

For details about this limitation with the Azure Load Balancer please see the Microsoft documentation. For details about this limitation with the AWS Network Load Balancer, please see Amazon's documentation.

As mentioned in the Summary, this only affects transparent TCP (layer 4) load balancers. Azure Load Balancers and AWS Network Load Balancers are two examples, but other load balancers may share this limitation.

The cause of the problem is that the Gorouter is initiating the call to the Route Service, which resides inside Cloud Foundry so the request goes to your Load Balancer and back to Gorouter. This in turn makes the source, Gorouter, and destination, Gorouter, the same.

Available solutions to this issue is to use either:

1. Don't use route services. Instead, you can use container to container networking. In this case, your route service would be an app deployed to the foundation and it would receive requests, apply logic, and conditionally route requests to a second app deployed to the same foundation and space over the container to container network. If you only have an internal route mapped to the second app, this clients from bypassing the first app which enforces the conditional access logic.
2. Deploy a second layer of layer 7 load balancers. In this setup, you would have traffic first go to the Layer 4 / TCP Load balancer, then to a Layer 7 HTTP Load balancer, like HAproxy, and then to Gorouter. This second layer of load balancers prevents Gorouter from being both the source and destination and avoids the limitation of the Layer 4 / TCP load balancers.
3. For Azure and AWS, you can use Azure app gateways or AWS application load balancers instead of the Layer 4/ TCP load balancer equivalent.
4. Use a static route service. In this scenario, described in the documentation, the route service resides outside of the TAS Foundation and so you don't requests looping from Gorouter directly back to your foundation's load balancers.