Slow in accessing websites due to policy trace rule without conditions

Products

• Advanced Secure Gateway Software - ASG
• ProxySG Software - SGOS

Issue/Introduction

User experience slowness issue when browsing to all sites.
The issue becomes more obvious when user traffic load is increasing.

There is no error message but user will notice the sites that they browse will be slow in loading the objects.

Cause

The packet capture will show the delay for the proxy to connect to the OCS at frame 218565.

No.     Time                        Source    Destination    SrcPort DstPort Protocol Length Info
 186013 2018-01-04 04:45:39.135326  x.x.x.x   y.y.y.y  36069   8080    TCP      66     36069 ? 8080 [SYN] Seq=0 Win=8192 Len=0 MSS=1340 WS=256 SACK_PERM=1
 186014 2018-01-04 04:45:39.135343  y.y.y.y   x.x.x.x  8080    36069   TCP      62     8080 ? 36069 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1460 SACK_PERM=1
 186019 2018-01-04 04:45:39.136006  x.x.x.x   y.y.y.y  36069   8080    TCP      60     36069 ? 8080 [ACK] Seq=1 Ack=1 Win=64320 Len=0
 186021 2018-01-04 04:45:39.136610  x.x.x.x   y.y.y.y  36069   8080    HTTP     268    CONNECT www.abc.com:443 HTTP/1.0
 186026 2018-01-04 04:45:39.137751  y.y.y.y   x.x.x.x  8080    36069   HTTP     93     HTTP/1.1 200 Connection established
 186034 2018-01-04 04:45:39.138493  x.x.x.x   y.y.y.y  36069   8080    TCP      60     36069 ? 8080 [ACK] Seq=215 Ack=40 Win=64281 Len=0
 186043 2018-01-04 04:45:39.139150  x.x.x.x   y.y.y.y  36069   8080    TLSv1.2  271    Client Hello
 186853 2018-01-04 04:45:39.225143  y.y.y.y   x.x.x.x  8080    36069   TCP      54     8080 ? 36069 [ACK] Seq=40 Ack=432 Win=65443 Len=0
 218565 2018-01-04 04:45:44.315881  y.y.y.y   z.z.z.z  59316   443     TCP      62     59316 ? 443 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 SACK_PERM=1
 
 x.x.x.x is the client IP
 y.y.y.y is the proxy IP
 z.z.z.z is the OCS IP


HTTP debug will show the delay prior to "POLICY: Evaluating PE_POLICY_CHECKPOINT_NEW_CONNECTION" process. The delay seeing in the HTTP debug (about 5 seconds) does correlate with the delay seeing in the packet capture.
 
1675.809 HTTP   Explicit HTTP INFO   HTTP SW 2FC2EB6B50 for 4E8C86CB Creating request, Full URL: http://www.abc.com
1675.808 HTTP   Explicit HTTP INFO   HTTP CW 4E8C86CB50              request from: x.x.x.x, URL: http://www.abc.com
1675.808 HTTP   Explicit HTTP NOTICE HTTP CW 4E8C86CB50              POLICY: Evaluating PE_POLICY_CHECKPOINT_NEW_CONNECTION
…
1670.402 HTTP   Explicit HTTP INFO   HTTP CW 4E8C86CB50              Decision on client connection is to persist.


When look at their policy, there is the following rules without any conditions:

;; Tab: [PolicyTrace]
   trace.request(yes)  trace.destination("http_policytrace") ; Rule 1
   trace.request(yes)  trace.destination("https_policytrace") ; Rule 2

Resolution

Remove or disable the rules that enabling policy tracing without any conditions.
Another option would be to put a condition like a client IP on the rules instead of having policy tracing for all the user request.

The proxy management GUI and the documentation did mentioned that enabling policy trace for all the request may cause a signifcant decrease in performance.