Interpreting web server mod_proxy

Products

Pivotal Web Server

Issue/Introduction

You see these errors or similar errors in the proxy server error log:

[error] (146)Connection refused: proxy: HTTP: attempt to connect to (backend ip:port/hostname) failed

[error] proxy: pass request body failed to (backend ip:port/hostname)

The clients originating failed requests receive 502, 400, or other error responses, usually after a lengthy wait.

Resolution

When troubleshooting, it is important to note the frequency of these messages and messages in the logs.

If every proxied request is failing to connect, the backend server is not running or a network misconfiguration is blocking access. You must test connectivity more directly by using a tool such as telnet to attempt to connect from the proxy to the backend. This can be used repeatedly to verify the problem and progress in fixing it.
If connection failures come in clusters, then a likely explanation is that the backend server is receiving more requests than it is able to accept (or configured to accept).
Sporadic connection failures may be signs of impending hardware failure.

Errors with proxy requests that are in flight or pass request body failed tend to be more subtle. Here are some things to try:

For HTTP proxies, try disabling reuse of proxy-to-backend connections (such as keepalive). You can do this by setting the proxy-nokeepalive variable in your configuration:
```
SetEnv proxy-nokeepalive 1
```

Note: If the problem disappears with this change, there is likely a subtle problem related to handling keepalive timeouts on one end of the connection or in between. Disabling keepalives may result in changes to proxy performance; you will need to test your particular systems to determine any impact. In most situations where proxy and backend are close on the network, the difference will be negligible, so it's best to make this change if it solves the problem.

Check the access log to see if errors are correlated with a particular pattern of request. For example, the backend may have an error and timeout when attempting a particular operation at a certain URL. Such a pattern generally indicates problems with the backend that should be researched there via logs and other debugging methods.

Important note: In all such cases, if the cause is not evident fairly quickly, you should use a tool like tcpdump, ethereal, or WireShark to capture a packet trace of the route between proxy and backend (preferably taken at both ends) to determine where the problem is. This should be used to determine the direction to look next.

If the proxy does not actually send the full request, focus on examining the client request and proxy configuration.
If the backend stops responding or closes the connection prematurely, then the backend should be investigated.
If there are many network errors, mysterious disconnections, or packets sent on one end that don't reach the other, then networking should be inspected.

Also make sure that you are using a recent version of the proxy server; proxy code has been a source of a few bugs. Check the specific version you are using against the Apache HTTP Server changelog (for 2.0 or 2.2 versions) to see if your problem has been previously described and fixed.