How to bypass a load balancer in TAS platform

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The Gorouters in TAS are always balanced by a load balancer. This could be the provided HAProxy appliance(s) or an external load balancer. Regardless, the load balancer VIP should be mapped to the api.domain in the environment. 

Q: When and why do we need to bypass the load balancer? 
A: When troubleshooting scenarios where an app is not accessible, cf cli commands do not work, there are unexpected HTTP responses, timeouts reached etc. it is always good practice to bypass the load balancer to verify if the issue is caused by the load balancer itself or if the issue is somewhere else on the datapath.

Resolution

To bypass the load balancer:

1. SSH to a jump box which has network access to the TAS deployed VMs.

2. Using curl, verify if there is a valid HTTP response from the api.domain.

curl -kv api.domain 

If this works, you should get a 200 OK or a 301 HTTP response. If you don't, this is an issue. Please proceed with the next steps. 

3. If you do not know the VIP of the load balancer you can use ping (if allowed by firewall), dig, nslookup etc. to find it out. 

Example: 

ping  api.domains.com 

or 

dig @<dns_server_ip> api.domain.com

or 

nslookup api.domain.com

4. Note down the IP to make sure later by comparison that we have successfully bypassed it. 

5. From bosh cli run bosh vms| grep -w "router" to get the IP address of any Gorouter. 

6. Using text editor, modify the file /etc/hosts on your jump box to add a manual entry for one of the Gorouters pointing to the api.domain. DNS lookup should use the local /etc/hosts file as a first source and if an entry is not available then it queries the DNS server. This means that the manual entry will overwrite the DNS record from the DNS server. 

Example:

• sudo vim /etc/hosts
• Press "i" to insert
• add an entry:
• <GorouterIP> api.domain.com
• Press Esc
• Press ":" followed by "wq" in order to write the change and quit.

7. Like in Step 3, check if the api.domain now resolves to the GorouterIP instead of the VIP of the load balancer. If it does, you have successfully bypassed the load balancer. 

8. Using curl, verify if there is a valid HTTP response from the api.domain.

curl -kv api.domain

If this works, you should get a 200 OK or a 301 HTTP response. 

• If the issue does not manifest after bypassing the load balancer, your problem is likely to be directly related to your load balancer and this component will need to be troubleshot by the managing team. 
• If the issue still exists, you can try to point to another Gorouter to make sure the problem is not in the Gorouter you have randomly picked. If you still have the problem after pointing to a second Gorouter, please engage Tanzu Support for further troubleshooting. 

Alternatively, you can send a curl request using a custom host header where you specify a Gorouter IP address. For this method, please follow this KB Article.