HTTP 500 errors may be seen on Avi VS when HTTP request policy is configured with Pool + Server content switch

Products

• VMware Avi Load Balancer

Issue/Introduction

Client traffic may experience unexpected HTTP 500 'Internal Server Error' responses from an Avi Virtual Service.

This issue occurs without any immediately obvious configuration errors and presents itself under the following specific conditions:

1. An HTTP Request Policy configured on the VS is using the Pool + Server content switch action. This issue will happen only if the HTTP policy is matched for that specific connection.

   1. Example screenshot of an HTTP rule which could cause the 500 error response -
   2. 

2. The Default Server Port in the associated Pool configuration is modified (for example, during a Disaster Recovery failover or routine maintenance).

   1. Screenshot of the Pool configuration field in question -
   2. 

Cause

The HTTP 500 error is caused by a server lookup failure during the content switching action.

While Avi has strict validation checks in place to prevent modifications to a server's IP address or port under the 'Servers' tab if that server is tied to an active HTTP policy, it currently lacks a similar validation check for the Default Server Port field at the broader Pool level.

Because the system permits the 'Default Server Port' to be changed, the specific server referenced in the HTTP Request Policy inadvertently enters an invalid or "bad" state. When traffic attempts to hit the policy, the server lookup fails, resulting in the 500 error.

Resolution

A permanent fix to address this validation gap will be included in future software versions.

Workaround:

• To prevent or immediately resolve this issue, do not explicitly specify or select individual pool servers within the content switch policy.
• Instead, configure the content switch action to route traffic strictly to the Pool, allowing the system's standard load-balancing algorithm to handle server selection.
• Example screenshot of the modified HTTP rule -
  • 
• If you absolutely need to select individual pool servers within the HTTP request rule, do not change the 'Default Server Port' field in the Pool configuration. If the change has already been made, revert this value to the original working port. 

Additional Information

Virtual service policies guide: https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/vmware-avi-load-balancer-configuration-guide/load-balancing-overview/virtual-services/virtual-service-policies.html