When a router redirects a packet to a cache/proxy for further processing, the cache/proxy will either terminate the session or bypass the session. If the Proxy terminates the session, it will route packets back to the client using via its routing table. Most often this is the default route in the Proxy. This can be, and many times is different from the interface on the router that originally redirected the packet.
Using the Return to Sender global switch at the cache/proxy will override the lookup in the routing table for the client return packet and sends the packet back to the MAC in the original packet (e.g. the router that redirected the packet via WCCP). If the WCCP router is a firewall (Cisco ASA routers) then it is likely that restrictions exist for inbound traffic. Thus, packets originating from the Proxy will have TCP ports that are blocked at this router and connections will not complete.
What to look for:
Get a PCAP of the initial client SYN (redirected, of course by WCCP).
When the Proxy sends a SYN/ACK to the client, there is nothing at all returned by the client.
Look at the Firewall logs to determine what packets are being dropped.
1) Change the Firewall rules to allow the SG to send packets back on all tcp ports
2) Do not use RTS and use a route or routes in the proxy to avoid the firewall for the client subnet(s)*
* Note that RTS inbound is enabled by default on Proxy (PR) license ProxySG devices, but disabled by default on ADN (M5) license ProxySG devices.
The return-to-sender setting is managed from the ProxySG Command Line Interface as follows:
#return-to-sender inbound (disable|enable|overwrite-static-route)