When the upstream network does not respond to the proxy - tcp_error

Products

Advanced Secure Gateway Software - ASG ProxySG Software - SGOS

Issue/Introduction

All of the below scenarios assume that a packet capture taken on the proxy shows SYN packets sent from the proxy to the server IP address, with no SYN,ACK responses.

Typically, the browser and/or event logs show a tcp_error, this means the upstream network is not responding to the proxy. This article elaborates on the following related article: Error: "Network error (tcp_error)" when browsing the Internet; 503 error returned to the client, focusing here on the situation where bypassing the proxy resolves the issue, which due to the superficial logic, tends to lead customers down the wrong line of investigation.

This article lists the most common causes of this, in order to help the customer to investigate in the right direction. This is important since in cases like this, the proxy is already doing all it can to make the connection. Even though this scenario very clearly indicates an issue with either the upstream network, or an upstream device or even the server itself, very often users are still able to access the site when bypassing the proxy, and so ignore the packet capture evidence and focus incorrectly on the proxy.

Resolution

All of these causes have been seen, some of them very often, in real-life production networks, and are worth investigating when the proxy gets no response to its SYN calls, even if bypassing the proxy resolves it.

1- Statistically, the most common point of failure is a firewall.

This is not an exhaustive list, but issues seen include:

a firewall blocking the port required, when the source is the proxy
a firewall blocking the proxy's IP address - this of course affects all web access via the proxy
any type of filtering or policy rule being hit on the firewall, including devices with application level awareness blocking traffic because of policy.
a firewall dropping traffic intermittently because of session limits, loading or timeout configuration. One example, from the link to the Cisco Forum, ASA seems to be dropping valid TCP SYN packets.
firewalls dropping traffic intermittently because of incorrectly configured security settings, such as port scan protection, DoS etc, dropping valid SYNs because they all come from the same IP, the proxy's IP.
firewalls being unable to handle http 1.1 persistence. More about this here: Upstream Firewall dropping connections from the ProxySG
firewalls not allowing path MTU discovery, as described in Why does disabling Path MTU discovery fix my connectivity problem?, bypassing may resolve because the client is not doing path MTU discovery.

2- Web server issues that are resolved when bypassing the proxy.

a server ACL. This could include a simple IP address ACL, especially for internal servers, but some also have blocks to limit access to certain geographic regions. Bypassing may work of course if the proxy's egress IP is in a different region to the client's network.
The Bluecoat ADN (Mach5 feature) serial number included as unknown bytes in the SYN packets could cause problems with some servers that assume this is malware.
as unlikely as this seems, we have seen cases where the server objects to the proxy's default X-Bluecoat-Via header, even though X-headers are only informational and RFC compliant. One server example is in our following KB article: Remove X-Bluecoat-Via Header

3- Routing problems on the network.

In explicit mode (the most common proxy deployment) the route out of the network may well be different compared to bypassing the proxy, and this could account for a failure of all access via the proxy and working when bypassing it. Even when the proxy is deployed transparently, if forwarding is configured and the forwarding host is not responding, this will have much the same result, more on this here in Error: 504 Gateway Error
In some cases, the server SYN,ACKs reach the network, then are lost because of asymmetrical routing, often caused by static routes on upstream devices.
We have seen cases where the upstream connectivity intermittently fails, with a proxy packet capture showing one single proxy SYN packet, then an upstream device (usually the gateway) returns an ICMP fail such as Time-to-live exceeded (Time to live exceeded in transit). The next request may work, and the network responds. The root cause was a software bug on a router.

4- Other upstream devices - web categorizing, filtering or scanning devices and agents can drop SYN packets at random due to software bugs.

Just two examples of many, here: