ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Increased Latency and Page Timeouts Occur in Proxy Forwarding to Web Security Service Deployment

book

Article ID: 174576

calendar_today

Updated On:

Products

Web Security Service - WSS ProxySG Software - SGOS

Issue/Introduction

In a deployment where the ProxySG appliance forwards traffic to the Symantec Web Security Service (WSS) and a large number of users use the service, latency increases and page-loading timeouts occur. These issues might occur because the pool of TCP connections is exhausted. 

Cause

The pool of TCP connections is exhausted because the ProxySG appliance is forwarding traffic to a single destination (WSS) rather than dispersing traffic to multiple public-content providers. Additionally, the pool of connections might also be exhausted by:

  • A large number of users using the WSS
  • Applications that consume a large number of TCP ports, such as Microsoft Office 365 applications

This article provides instructions on how to detect and resolve this situation. 

Resolution

Determine If All Connections Are Being Consumed

To determine if all connections are being consumed, view the statistic for the ProxySG appliance that increases when the appliance was unable to find a source port:

  1. In a web browser, go to:
    https://<ProxySG_IP_Address>:<Port_Number>/TCP/Statistics?stats_mode=3
  2. Locate the row for TCP2.214. Ensure the number for this protocol remains static and does not increase as peak load occurs. A number that increases means that the appliance was not able to find a source port. 

Resolution: Increase the Number of Available Connections

To increase the number of available connections, do the following until the TCP2.214 statistic remains static, and the latency and timeout issues are resolved:

If the issues persist, contact Symantec support for further investigation.

Increase Maximum Number of Source Ports for the ProxySG Appliance

By default, the maximum number of source ports is 16,384. To ensure enough unique connections are available, use the following CLI command to increase the number of ports to the maximum possible for the appliance:

#(config) tcp-ip inet-lowport 16384

Note: The inet-lowport can be set as low as 1024. Setting the port lower than a listening port on the proxy can had adverse effects to regular proxy operations.

For further information, see the KB article: https://knowledge.broadcom.com/external/article?articleId=167384

Reduce the Amount of Time a TCP Connection is in the TIME_WAIT State

To reduce the amount of time a TCP connection is in the TIME_WAIT state, use the following CLI command:

#(config) tcp-ip tcp-2msl 30

Note: For other devices in your network, such as firewalls, ensure that the TCP TIME_WAIT state value is not lower than the value you set for the ProxySG appliance. 

Reducing the TCP TIME_WAIT state value ensures that the ProxySG source ports become reusable more quickly.

Configure Additional Egress IP Addresses from Your ProxySG Appliance to the Web Security Service

To configure additional egress IP addresses from your appliance to WSS:

  1. Configure additional IP addresses by using the following CLI command:
    #(config interface <interface_number>) ip-address <ip-address> <subnet-mask>
  2. Create policy to divide outgoing connections between the additional IP addresses. To do this, configure client subnets that are approximately the same size to use different addresses. The following is an example of CPL to configure the subnets:
    <forward>
    client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_one>)
    client.address=<ip_address_of_client_subnet> reflect_ip(<sg_ip_two>)

    and so on for as many client subnets as necessary.
  3. Save and deploy policy.

Configure the ProxySG Appliance to Connect to Multiple Web Security Service IP Addresses (Limited Availability)

Note: This option is available on a limited basis in some data centers.

Use this option if you want to avoid adding public IP addresses to your ProxySG appliance.

To obtain additional IP addresses for WSS, contact your Symantec point of contact for assistance. When you have the additional IP addresses, do the following:

Set Up Forwarding Hosts for Each IP Address

To set up forwarding hosts:

  1. In the Management Console, select the Configuration > Forwarding > Forwarding Hosts tab.
  2. Click New.
  3. Configure the host options appropriately for each IP address. For information on configuring host options for each WSS port, see steps 3, 4, and 5 of the following document:
    https://portal.threatpulse.com/docs/sol/connectivity/fixed/proxy/conn-prxyfwd-symapp.htm
    Note: When creating aliases, ensure you create a unique alias for each forwarding host you create.
  4. Repeat these steps for each IP address.
    Note: If you are configuring multiple ports per IP address, you will need to repeat these steps multiple times per IP address. For example, if you want to configure a host for ports 8080 and 8443 for an IP address, create two hosts for that address; one host for port 8080 for that address and another host for port 8443 for the same address.

Create a Forwarding Group

Create forwarding groups for each WSS port you configured a forward host for. For example, if you created forwarding hosts for ports 8080 and 8443, then you will create two forwarding groups, one for each port.

To create a forwarding group:

  1. In the Management Console, select the Configuration > Forwarding > Forwarding Groups tab.
  2. Click New.
  3. In the Alias field, enter a unique name for the forwarding group.
    Note: Because the forwarding group alias is used in policy, the alias cannot be a CPL keyword, such as no, default, or forward.
  4. In the Alias name field, select the hosts you previously created and click Add.
  5. In the Load Balancing and Host Affinity section, select the following:
    • From the Load balancing method list, select Least Connections.
    • From the Host affinity methods list, select Client IP Address for all applicable host affinities.
  6. Click OK.
  7. Click Apply.
  8. Repeat these steps for each WSS port you created a forwarding host for.

(Optional) Edit the Health Check for the Forwarding Group

To edit the health check for your forwarding group:

  1. In the Management Console, select the Configuration > Health Checks > General.
  2. Select the health check for the forwarding group you want to edit.
  3. Click Edit.
  4. In the Minimum number of members that must be healthy for the group to be healthy dropdown, select either All or Any.

Configure Policy to Use the Forwarding Groups

Using either the CPL or the VPM, configure policy to reference the appropriate forwarding group names. For information on proxy forwarding policy, see: https://portal.threatpulse.com/docs/sol/connectivity/fixed/proxy/conn-fwdpolicy.htm

Verify Load Balancing is Functioning

Verify that your ProxySG appliance balances traffic equally among the WSS IP addresses.

To verify if load balancing is functioning:

  1. In a web browser, go to:
    https://<ProxySG_IP_Address>:<Port_Number>/Forwarding/StatsIP
  2. View the Connect Active (Total) statistic for each IP address in the forwarding groups you created to resolve the connection issues. If load balancing is functioning correctly, the number of connections for each address should be similar.