Increased Latency and Page Timeouts in Proxy Forwarding to Cloud SWG
search cancel

Increased Latency and Page Timeouts in Proxy Forwarding to Cloud SWG

book

Article ID: 174576

calendar_today

Updated On:

Products

Cloud Secure Web Gateway - Cloud SWG ProxySG Software - SGOS

Issue/Introduction

This article outlines detection and resolution steps for connection exhaustion when forwarding traffic from a Edge SWG appliance to Cloud SWG.

Symptoms:

  • Increased latency to websites and page timeouts.
  • Intermittent browsing failure or a perceived "system down" state, particularly during peak business hours.
  • The frequency correlates with local proxy load. Issues occur primarily during peak usage and are absent during low-traffic periods, such as early mornings, late evenings, weekends, or holidays.
  • HTTP 503 (Service Unavailable) client responses occurring when the proxy drops packets due to port reuse.
  • Health check timeouts and periodic failures on the forwarding proxy.
  • No performance degradation announcements on the Cloud SWG Status page for the relevant data center.

Environment

Proxy forwarding to Cloud SWG (formerly WSS)

Cause

The pool of TCP connections is exhausted because the Edge SWG appliance is forwarding traffic to a single destination (Cloud SWG) rather than dispersing traffic to multiple public-content providers. Exhaustion is often accelerated by:

  • Large numbers of users accessing Cloud SWG applications that consume high volumes of TCP ports (e.g., Microsoft Office 365).
  • Inefficient HTTP Persistence: Disabling HTTP persistence globally increases the demand for new source ports.

Resolution

Determine If All Connections Are Being Consumed

To determine if all connections are being consumed, monitor the Edge SWG statistic that tracks failures to find a source port: 

  1. Navigate to: https://<ProxySG_IP_Address>:<Port_Number>/TCP/Statistics?stats_mode=3
  2. Locate the row for TCP2.214.
  3. Verify that this protocol remains static. An increasing count indicates the appliance is unable to find an available source port during peak load.  

Resolution: Increase the Number of Available Connections

Perform the following steps until the TCP2.214 statistic remains static and performance stabilizes. Contact Symantec support if the issue persists. 

Increase Maximum Number of Source Ports for the Edge SWG Appliance

By default, the maximum number of source ports is 16,384. To ensure enough unique connections are available, use the following CLI command to increase the number of ports to the maximum possible for the appliance:

#(config) tcp-ip inet-lowport 16384

Note: The minimum inet-lowport value is 1024. Setting a value lower than a proxy listening port may adversely affect operations.

For further information, see the KB article: https://knowledge.broadcom.com/external/article?articleId=167384

Reduce the Amount of Time a TCP Connection is in the TIME_WAIT State

To reduce the amount of time a TCP connection is in the TIME_WAIT state, use the following CLI command:

#(config) tcp-ip tcp-2msl 30

Note: For other devices in your network, such as firewalls, ensure that the TCP TIME_WAIT state value is not lower than the value you set for the ProxySG appliance. 

Reducing the TCP TIME_WAIT state value ensures that the Edge SWG source ports become reusable more quickly.

Configure Additional Egress IP Addresses from EdgeSWG Appliance to the Cloud SWG

If port exhaustion continues, configure additional egress IP addresses on the ProxySG to provide more unique source port combinations:

  1. Add IP Addresses:
    #(config interface <interface_number>) ip-address <ip-address> <subnet-mask>
  2. Create Policy: Use CPL to divide outgoing connections between the additional addresses by client subnet.
    <forward>
    client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_one>)
    client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_two>)
  3. Save and deploy the policy.

Configure the Edge SWG Appliance to Connect to Multiple Web Security Service IP Addresses (Limited Availability)

Note: This option is available on a limited basis in some data centers.

This option avoids adding public IP addresses to the ProxySG. Contact a Symantec representative to confirm availability for the relevant data center and to obtain additional Cloud SWG IP addresses.

Set Up Forwarding Hosts for Each IP Address

To set up forwarding hosts:

  1. In the Management Console, select the Configuration > Forwarding > Forwarding Hosts tab.
  2. Click New.
  3. Configure the host options appropriately for each IP address. For information on configuring host options for each Cloud SWG port, see steps 3, 4, and 5 from "Procedure—Configure the Appliance" of the following document:
    https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/web-security-service/help/conn-matrix/conn-about-proxyforward/conn-prxyfwd-symapp.html
    Note: When creating aliases, ensure you create a unique alias for each forwarding host you create.
  4. Repeat the process for each IP address.
    Note: If you are configuring multiple ports per IP address, you will need to repeat these steps multiple times per IP address. For example, if you want to configure a host for ports 8080 and 8443 for an IP address, create two hosts for that address; one host for port 8080 for that address and another host for port 8443 for the same address.

Create a Forwarding Group

Create forwarding groups for each configured Cloud SWG port. For example, if you created forwarding hosts for ports 8080 and 8443, then you will create two forwarding groups, one for each port.

To create a forwarding group:

  1. In the Management Console, select the Configuration > Forwarding > Forwarding Groups tab.
  2. Click New.
  3. In the Alias field, enter a unique name for the forwarding group.
    Note: Because the forwarding group alias is used in policy, the alias cannot be a CPL keyword, such as no, default, or forward.
  4. In the Alias name field, select the previously created and click Add.
  5. In the Load Balancing and Host Affinity section, select the following:
    • From the Load balancing method list, select Least Connections.
    • From the Host affinity methods list, select Client IP Address for all applicable host affinities.
  6. Click OK.
  7. Click Apply.
  8. Repeat these steps for each Cloud SWG port with a created forwarding host.

(Optional) Edit the Health Check for the Forwarding Group

To edit the health check for your forwarding group:

  1. In the Management Console, select the Configuration > Health Checks > General.
  2. Select the relevant health check for the forwarding group.
  3. Click Edit.
  4. In the Minimum number of members that must be healthy for the group to be healthy dropdown, select either All or Any.

Configure Policy to Use the Forwarding Groups

Using either the CPL or the VPM, configure policy to reference the appropriate forwarding group names. For information on proxy forwarding policy, see: https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/web-security-service/help/conn-matrix/conn-about-proxyforward/conn-fwdpolicy.html

Verify Load Balancing is Functioning

  • Navigate to: https://<ProxySG_IP_Address>:<Port_Number>/Forwarding/StatsIP
  • Verify that the Connect Active (Total) statistics are distributed similarly across all IP addresses in the forwarding group.
Powered by Wolken Software