Troubleshooting 5xx responses received from Avi Load balancer

Products

• VMware Avi Load Balancer

Issue/Introduction

The application shows a 500/503 response code when hosted on the Avi load balancer. 

 

Cause

There could be multiple configuration points that can lead to a 500 or 503 response from the Avi Load Balancer. 

A few of them are listed below with concerning logs suggesting the cause:

1. Validate if the server is sending a 503 or 500 response.

   • Check the same from VS Logs.
     Significance log: "Request ended abnormally:response code 5xx"
     Following is a sample snippet: 
     
     

     For such behavior, check the backend application server to identify the cause of such response.

2. SSL handshake failure with the server.

   • Check the VS logs on UI with significance log: "Connection closed abnormally: connection dropped due to server side SSL handshake failure"
     A sample screenshot is below :
     
     
     
     

     To resolve the above, consider checking the two points below:
     
     

     • Check if the pool is configured with SSL settings - Edit pool > ssl.
       If the pool is configured for SSL, the server(s) should be capable of handling SSL connections. If that is not the case, please remove any SSL configuration in the pool settings.
       
       Reference document for SSL configuration on pool: SSL Configuration for Pool

      

     • If the server is expected to handle SSL connection, please contact Broadcom technical support to help identify where the SSL handshake is failing.

      

3. Unable to connect to the backend server on the configured port.

   • The above can be identified from VS logs with significance log : "Connection setup failed: server conn setup timed out".
     A sample screenshot is below:

     
     
     

     
     For pools with no health monitor configured. the backend server will not be marked down when unreachable on the desired port. As the Avi service engine is not receiving any response from the server, Avi LB vip will respond with a 503 response code to the client.
      

     To resolve the above behavior, consider checking the below points:
      

     • Check if the application is available at the configured port. Accessing the server IP directly (bypassing LB VIP), if possible, can help identify the same easily.
     • Check if the Avi Service engine can connect to the server on the configured port. To validate, please attach a TCP health monitor to the pool. Please note that the port configuration on the health monitor should be the same as that of the pool. If the server is marked down after the health monitor is attached to the pool, check the reason for the same under Pool events and proceed accordingly with the next steps to fix the connectivity issue in the infrastructure. Below is a sample event for a Down pool member : 
       

      

      

4. The configured datascript on the virtual service is not correct

   • This can be identified from VS logs with significance: "Request ended abnormally: Datascript failed to execute"
     Below is a sample snippet:
     
     
     

     To resolve the above, please review the datascript attached to the virtual service and correct it as required.

     Reference guide for datascript functions supported by Avi load Balancer: Datascript functions

      

Resolution

Configuration on Avi virtiual service or pool should be corrected as required.