NSX T Native Load Balancer: LbPoolStatus is DOWN. Members: 'PoolMemeber_IP_Address:PortNumber' with status: 'The received content is not matched.'

Products

• VMware NSX

Issue/Introduction

Virtual Server is 'Down' and we can check the alarm associated with the specific virtual server as shown below for more details on the issue.\

Alarm:
Consult the load balancer pool to determine its status and verify its configuration. It incorrectly configured, reconfigure it and remove the load balancer pool from the virtual server then re-add it to the virtual server again.

Cause

The cause of the NSX T Virtual Servers being down can be identified as follows:

• Select the Virtual Servers.
• Click on 'Down' to view more details.
  • Errors: 

    LbVirtualServerStatus is DOWN

    •  

NSX-T L7 Virtual Servers are down because all the associated Server Pool Members has been detected as Down. The reason for this can be identified as follows:

• Select the Virtual Server Pool Member.
• Click on 'Down' to view more details.
  

  • Errors: 

    LbPoolStatus is DOWN. Members: 
    'PoolMemeber_IP_Address:PortNumber' with status: 'The received content is not matched.'

    • 
  • If you check the server pool alarms, you may notice the following or similar descriptions and recommendations:
    • 

Based on the information above, we observe that the server is still reachable; however, it is not sending the response that you have configured, either from the default or custom profile, as an acceptable response to confirm that the server is functioning as expected.

Resolution

In an NSX-T environment, when a active monitor is configured to check the health of a server or virtual machine (VM) in a pool, it typically expects specific response codes to confirm the server's health and availability. These response codes are crucial for the load balancer to assess whether a server is functioning properly and capable of handling incoming traffic. If there’s a mismatch between the expected and actual response code, the server will be marked as "Down". For example, say HTTP health check is configured to expect a specific response code, such as 200. Meanwhile, the server is sending back a different code, such as 404, 500 (Internal Server Error) or 503 (Service Unavailable) then that server would be marked as down

Note: Once the server is marked down, the load balancer will route incoming user requests to the remaining healthy servers in the pool, bypassing the problematic server. This ensures that service availability is maintained even if one server is not functioning correctly.

Troubleshooting:

• Coordinate with the server administrator to verify if there are any server side application/service issues by accessing the server's IP or FQDN directly, bypassing the load balancer. Additionally, review the server logs and allow the server admin to continue troubleshooting to confirm the server’s health.
• If there are new or different response codes from the server, they need to be added to the monitor's list of acceptable response codes accordingly. Please refer the screen shot for reference,
  • Using the "Monitors" and "Profiles" create custom profile and link that to the active monitor configured for the virtual server.
    • 
  • Refer to the below screen shot where in a custom monitor profile, when you create a custom response codes.
    • 
    • 
  • You can disable this active monitor temporarily or permanently, based on your needs, and choose to use the NSX-T load balancer without the active monitor.
    
    • 
    • Click "Edit" as shown above
    • 
    • Then click "1" beside the Active Monitor as shown above
    • 
    • Uncheck whatever is selected and save.
    • This will remove the active monitor from the profile, meaning it will no longer monitor the service based on the expected response code. Instead, it will only trigger an error if the server is unreachable via the network or port. Refer the following KB for more information on Troubleshooting NSX Native Load Balancer