Performance and connectivity errors accessing web site through dedicated IP address

Products

• SG-S500
• Cloud Secure Web Gateway - Cloud SWG

Issue/Introduction

Users accessing internet sites via Cloud SWG using Proxy forwarding access method without issues.

Dedicated IP address feature enabled for organisation's intranet site.

After making PAC file changes to send 100k users to intranet site via Cloud SWG, many users reported slowness rendering pages or timeout errors.

Environment

Proxy forwarding.

Cloud SWG.

Dedicated IP address.

Cause

Intranet web site closing connections after idle timeout without any indication to Cloud SWG.

Resolution

Multiple solutions exist to address the issue on OCS side, although final change was to the on premise proxy:

• Increase the idle session timeout parameter of 120 seconds configured on the OCS side: Any TCP session that was idle for 120 seconds was removed from the ICS side, without any corresponding TCP FIN or RESET being seen by the Cloud SWG side.
• Modify the algorithm on OCS side so that any TCP connections exceeding the idle session timeout would be TCP FIN'd, requiring an ACK from the Cloud SWG side and a corresponding FIN: This could notify the Cloud SWG stack to close and re-issue the request in a new TCP session 
• Modify client persistent timeout on the on-premise proxy (https://knowledge.broadcom.com/external/article/166251/edge-swg-proxysg-timeout-value-when-comm.html) so that it is LESS than that of the OCS: In our case, we dropped the timeout to 90 seconds. When that timeout was exceeded, the on-premise proxy would TCP FIN the upstream and downstream connection, and the client would never re-use the existing TCP port to hit the erroneous use case above.

It was decided to go with the last option of modifying the on-premise idle client timeout and no further issues were seen.

 

Additional Information

From the PCAPs taken on the dedicated IP address gateways, we can see that everytime the problem happened, the existing socket was re-used after a certain time (243 seconds in this case):

In this case, the request would be retransmitted 10 times before a TCP FIN was issued, which typically took about 3 minutes. 

Since the Cloud Proxy did not get a response back to it's request within 180 seconds, the TCP_ERROR was triggered.