Web page delay and 503 HTTP response due to the combination of TCP retransmission sent by SG and security function by firewall when reflect_client_ip is enabled

book

Article ID: 167730

calendar_today

Updated On:

Products

ProxySG Software - SGOS

Issue/Introduction

This problem occurs when the ProxySG is deployed as a transparent proxy and reflect_client_ip is enabled for HTTP interception.  Here is a sample packet capture of the problem:

=======================================================================
 Sample packet capture
=======================================================================
No.  Time    Source         Destination    S.Port D.Port Protocol Info
  1  2.59    [ Client IP ]  [Web  Server]  1702   80     HTTP     GET /xxx/test.htm HTTP/1.1
  2  2.65    [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
  3  2.68    [Web  Server]  [SG  IP Addr]  80     19109  TCP      80 > 19109 [SYN, ACK]
  4  2.68    [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [ACK]
  5  2.68    [SG  IP Addr]  [Web  Server]  19109  80     TCP      GET /xxx/test.htm HTTP/1.1
  6  2.88    [SG  IP Addr]  [Web  Server]  19109  80     TCP      [TCP Retransmission] GET /xxx/test.htm HTTP/1.1
  7  3.23    [SG  IP Addr]  [Web  Server]  19109  80     TCP      [TCP Retransmission] GET /xxx/test.htm HTTP/1.1
  8  3.26    [Web  Server]  [SG  IP Addr]  80     19109  TCP      80 > 19109 [RST]   <<---------- RST by Web server !!
  9  3.26    [SG  IP Addr]  [Web  Server]  19109  80     TCP      [TCP Port numbers reused] 19109 > 80 [SYN]
 10  6.23    [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 11  9.43    [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 12  12.63   [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 13  15.83   [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 14  19.03   [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 15  25.23   [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [SYN]
 16  25.26   [Web  Server]  [SG  IP Addr]  80     19109  TCP      80 > 19109 [SYN, ACK]
 17  25.26   [SG  IP Addr]  [Web  Server]  19109  80     TCP      19109 > 80 [ACK]
 18  25.26   [SG  IP Addr]  [Web  Server]  19109  80     TCP      GET /xxx/test.htm HTTP/1.1
 19  37.44   [SG  IP Addr]  [Web  Server]  19109  80     TCP      [TCP Retransmission] GET /xxx/test.htm HTTP/1.1
 20  101.44  [SG  IP Addr]  [Web  Server]  19109  80     TCP      [TCP Retransmission] GET /xxx/test.htm HTTP/1.1
 21  101.46  [Web  Server]  [SG  IP Addr]  80     19109  TCP      80 > 19109 [RST]   <<---------- RST by Web server !!
 22  101.46  [SG  IP Addr]  [ Client IP ]  80     1702   HTTP     HTTP/1.1 503 Service Unavailable  (text/html)

From the above sample Packet capture trace, the ProxySG will use same TCP source port (TCP port "19109") for retransmission (Frame#9-21).  There is a TCP/IP stack limitation with the reflect_client_ip setting such that SGOS does not have a dynamic way of choosing a new connection to establish after one has been closed (like RST as above Frame#8).  The ProxySG must re-establish the TCP source port.

 

Resolution

The problem does not occur when reflect_client_ip is disabled. The ProxySG will use a different TCP source port for frames 9-21 with source IP address as the ProxySG appliance's interface address.  Additionally when reflect client IP is enabled, there is a delay (see frames 9-16 when a SYN-ACK is returned from the web server) of around 20 seconds. This is caused by the firewall.

If there is Juniper ISG acting as a firewall between the ProxySG and the Web server, the TCP RST and initial SYN packet in the TCP three-way handshake can occur very quickly.  The TCP RST packet will cause a session to be marked as garbage and any new TCP connections on the same source port could result in failure to establish a connection.

Several firewall vendors have implemented this way of managing TCP sessions after a TCP reset is sent. Please contact the firewall vendor's support to ask about their design.  If a Juniper ISG is used, setting the "set flow tcp-rst-invalid-session" ScreenOS command could be used as a workaround for this case.