Load Balancer error logs regularly report - 'epoll_wait() reported that client prematurely closed connection'

Products

• VMware NSX

Issue/Introduction

• NSX Edge Load Balancer /var/log/lb/<lb-uuid>/error.log reports logs similar to the following:
  epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while reading upstream, client: <Application VM IP Address>
epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream, client: <Application VM IP Address> 
  
  
• Application communicating via NSX Native Load Balancer experiences regular intermediate timeouts. The timing of the logs appears to be an indicator for why a client application reports random timeouts, when requesting or changing data on pool members.

Environment

• VMware Cloud Foundation (VCF)
• VMware NSX
• NSX Native Load Balancer

Cause

This issue typically occurs when the client application or an intermediate network device sends a TCP Reset (RST) or FIN packet before the Load Balancer has completed the transaction. In NSX-T environments, this is often related to:

1. Mismatched Keepalive timeouts between the Load Balancer and backend servers.
2. Client-side timeouts occurring before the backend responds.
3. Network MTU mismatches causing packet drops and subsequent resets.

Resolution

1. Gather simultaneous packet captures from the Client VM, NSX Edge (Ingress/Egress), and Backend Server\Pool Member to identify which entity is initiating the premature TCP Reset. Refer to KB 341568 for details on how to use the pktcap-uw packet capture tool.
   • If it is the client VM then investigate further with your VM OS support and application teams.
   • If it is the Backend Server then investigate further with the owners of those servers. 
2. If the Load Balancer is sending a 502 Bad Gateway, verify if the backend server is closing the connection immediately before a new request arrives. Refer to KB 383374 for keepalive timeout recommendations.
3. Check the MTU settings across the logical and physical network path. Ensure the MTU is consistent (typically 1500 or 9000 for Jumbo Frames) and accounts for Geneve encapsulation overhead (minimum 1600 recommended for transport zones). Refer to the KB 374882 to troubleshoot MTU settings on the NSX Fabric.

Additional Information

• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.
• To add or edit an application profile on an NSX Native Load Balancer to control Load Balancer timeouts, refer to this Technical Document.