Interpreting web server mod_proxy

Products

VMware Support Only for Apache HTTP

Issue/Introduction

Symptoms:

You see these errors or similar errors in the proxy server error log:

[error] (146)Connection refused: proxy: HTTP: attempt to connect to (backend ip:port/hostname) failed

[error] proxy: pass request body failed to (backend ip:port/hostname)
The clients originating failed requests receive 502, 400, or other error responses, usually after a lengthy wait.

Environment

VMware vFabric Web Server 5.0
VMware vFabric Enterprise Ready Server 4
Apache HTTP Server 2.2
Apache HTTP Server 2.0

Cause

The error Connection refused indicates that the proxy cannot connect to the backend server to which it attempts to relay the request. There are several possible causes for this issue:

The backend web server is not responding, either because it is:
- not running.
- overwhelmed with requests and not accepting further connections (most likely explanation).
There is a networking problem between the proxy and backend. For example, a faulty interface configuration, faulty network devices, or a firewall. The Connection refused error is different from a complete failure to locate the backend (due to routing errors or missing DNS entries, for example). It means the backend host can be reached but cannot accept a connection at the specified port.

The error pass request body failed indicates that the proxy made a successful connection to the backend server and sent the request headers (which generally fit in the first packet). But, the connection for the body of the request (generally a POST request) is either dropped, or the backend server stopped reading the incoming data. There are several possible causes for this issue:

The backend web server crashed or hung while reading the request.
The backend web server stalled and timed out while processing data from a long body, so the request could not be completed.
The connection was closed prematurely, perhaps due to faulty networking or buggy implementations.

Resolution

When troubleshooting, it is important to note the frequency of these messages and messages in the logs.

If every proxied request is failing to connect, the backend server is not running or a network misconfiguration is blocking access. You must test connectivity more directly by using a tool such as telnet to attempt to connect from the proxy to the backend. This can be used repeatedly to verify the problem and progress in fixing it.
If connection failures come in clusters, then a likely explanation is that the backend server is receiving more requests than it is able to accept (or configured to accept).
Sporadic connection failures may be signs of impending hardware failure.

Errors with proxy requests that are in flight or pass request body failed tend to be more subtle. Here are some things to try:

For HTTP proxies, try disabling reuse of proxy-to-backend connections (such as keepalive). You can do this by setting the proxy-nokeepalive variable in your configuration:

SetEnv proxy-nokeepalive 1

Note: If the problem disappears with this change, there is likely a subtle problem related to handling keepalive timeouts on one end of the connection or in between. Disabling keepalives may result in changes to proxy performance; you will need to test your particular systems to determine any impact. In most situations where proxy and backend are close on the network, the difference will be negligible, so it's best to make this change if it solves the problem.

Check the access log to see if errors are correlated with a particular pattern of request. For example, the backend may have an error and timeout when attempting a particular operation at a certain URL. Such a pattern generally indicates problems with the backend that should be researched there via logs and other debugging methods.

Important Note: In all such cases, if the cause is not evident fairly quickly, you should use a tool like tcpdump, ethereal, or WireShark to capture a packet trace of the route between proxy and backend (preferably taken at both ends) to determine where the problem is. This should be used to determine the direction to look next.

If the proxy does not actually send the full request, focus on examining the client request and proxy configuration.
If the backend stops responding or closes the connection prematurely, then the backend should be investigated.
If there are many network errors, mysterious disconnections, or packets sent on one end that don't reach the other, then networking should be inspected.

Also make sure that you are using a recent version of the proxy server; proxy code has been a source of a few bugs. Check the specific version you are using against the Apache HTTP Server changelog (for 2.0 or 2.2 versions) to see if your problem has been previously described and fixed.