High CPU usage in SSL and Cryptography on Edge SWG
search cancel

High CPU usage in SSL and Cryptography on Edge SWG

book

Article ID: 168508

calendar_today

Updated On:

Products

Advanced Secure Gateway Software - ASG ProxySG Software - SGOS ISG Proxy

Issue/Introduction

CPU Monitor for ProxySG or Advanced Secure Gateway (ASG) is reporting high CPU utilization in SSL and Cryptography.

Resolution

High CPU being reported on the ProxySG/ASG can be due to one or more of the reasons which are listed below, or something else entirely. However, the most common reasons for high CPU in SSL and Cryptography are as follows:

High SSL traffic to specific sites

To determine if there is something in the network causing continuous policy evaluations, Blue Coat Reporter can be leveraged. The following Reporter reports can be helpful to further diagnose the issue:

  • SSL access logs show which protocols of TCP for SSL traffic is being used (e.g. tunneled). The SSL Access Log contains protocols of type TCP and SSL. The SSL protocol has been handed off to the SSL Proxy. 
  • Web requests per client IP report filter further based on the site and protocol. 

Additionally, the Blocked web browsing per user and Default Bandwidth reports are also useful. 

If after reviewing the access logs significant ssl traffic to specific sites is seen, apply a policy to disable SSL interception for a particular site(s). 
For example, to disable SSL interception for Facebook, Google, and Microsoft apply the following policy (for explicit deployment):

<proxy>
condition=SSL_Disabled_Domains detect_protocol(none)
 
define url.domain condition SSL_Disabled_Domains
facebook.com 
google.com
microsoft.com
end

A high number of requests from one or more clients

High CPU utilization can result when one or more clients send a high number of requests to the proxy. Examples include opening many simultaneous sessions to the same destination, or applications that do not understand proxy authentication leading to a loop of requests or authentication prompts.


To identify this situation, use the management console. Navigate to Statistics > Sessions > Active Sessions and see if a single user tries to make a large number of connections to the same destination. One possible resolution for this situation is to enable the Attack Detection feature by issuing the following commands at the CLI:

proxy>enable
proxy#conf t
proxy#(config)attack-detection
proxy#(config attack-detection)client
proxy#(config client)enable-limits

 

These commands limit the number of sessions that are permitted from any client up to 100 simultaneous sessions and subsequent sessions for this client IP are dropped. Care should be used when implementing this policy as it may prevent legitimate applications from connecting correctly to the Internet.

Note: This configuration is not recommended to be done globally on the proxy in case of:

  1. This proxy is used as upstream proxy for other proxies
  2. Group of clients connect to the proxy using the same IP as they come from any Network Address Translation (NAT) device.

Multiple certificates using the same Common Name

Beginning in SGOS 6.5.9.9 and 6.6.4.3 a fix was introduced to address this issue.  

In SGOS releases before 6.5.9.9 and 6.6.4.3 an issue can occur where multiple certificates use the same common name. This situation causes the ProxySG to continuously recreate its version of the certificates which can be a CPU-intensive process. Only if you are unable to upgrade to the latest version, the following workaround can be used. Change the SSL forward proxy policy in the VPM:

  1. Access the SSL Intercept layer.
  2. Edit the rule that is set to intercept SSL (right-click and select Edit where the action contains Enable HTTPS Interception Object).
  3. Select Splash Text and enter the following in the text box below it: $(x-rs-certificate-serial-number)$(x-rs-certificate-valid-from)$(x-rs-certificate-valid-to)
  4. Click OK > Install Policy.
If the appliance has the SSL interception enabled in CPL format only (instead of VPM), change the intercept line by adding the splash text as shown in this example:
<ssl-intercept>
ssl.forward_proxy(https) ssl.forward_proxy.splash_text("$(x-rs-certificate-serial-number)$(x-rs-certificate-valid-from)$(x-rs-certificate-valid-to)")

You can monitor the cert cache by accessing the following URL on the ProxySG appliance: https://<PROXYIP>:8082/sslproxy/certcache

Increase the Certificate Cache

To get the most out of the changes if the proxy is running SGOS 6.5.5.1 or later, increase the certificate cache timeout from the default of 2 hours to 72 hours, from the CLI:

proxy>enable
proxy#conf t
proxy#(config)ssl
proxy#(config ssl)proxy set-cert-cache-timeout 96

Lower the emulated certificate key size

With SSL-Interception, the ProxySG emulates original certificates from the websites and uses the configured key size to generate them.

Decrease the size of the ssl-interception private keys from the CLI:

proxy>enable
proxy#conf t
proxy#(config)ssl
proxy#(config ssl)proxy force-emulated-cert-keysize 2048

Untrusted Certificate on the Upstream

On a Proxy Sysinfo or Snapshot, you may see something similar to the following lines flooding the file:

2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: instance-2"  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746

This indicates that the Edge SWG is not trusting certificates from the upstream devices (reasons include certificates being expired or common-name mismatches, or other possible causes).

Workaround:

- Apply the following CPL:

<SSL>
server.certificate.validate(no)             
 
- Create the following rule on the VPM, as shown in Bypass server certificate validation using the ProxySG Visual Policy Manager:
  1. Create a Web Access Layer.
  2. Right-click Action > Set > New > Set Server Certificate Validation.
  3. Check "Disable Server Certificate Validation". Click OK and "Install Policy".

You should see the CPU utilization lower drastically within a few minutes.

Note:

Disabling certificate validation makes the ProxySG vulnerable to "Man In the Middle" attacks.

Solution:

In order to fix this issue permanently, identify the upstream devices causing the problem and troubleshoot the certificate issues on them.

To help during the troubleshooting process, reproduce the issue on a maintenance window (or before applying the fix) and:

  1. Take a Packet Capture
  2. Export the Active and Errored Sessions (Statistics > Sessions).
  3. Export the SSL Access Log.

Once you have identified the destination(s) of the errors, you may apply the rule above to only those destinations, while the problematic certificates are being fixed.

DHE cipher introduced in SGOS 6.5

DHE ciphers are disabled by default. ECDHE ciphers were added to provide a similar level of encryption with fewer CPU resource requirements. For more details on the ciphers shipped with SGOS, please refer to Cipher support on ProxySG.

proxy>enable
proxy#conf t
proxy#(config)ssl
proxy#(config ssl) proxy dhe-ciphers disable

Next Steps

If you go through these steps and still have issues with high CPU utilization in the HTTP or FTP process group, open a ticket with Broadcom Support.

In addition to the details from the CPU Monitor, you may also be asked to provide the following:

SysInfo

  • The SysInfo information should be captured after the CPU utilization has returned to normal, or after 20 minutes of high utilization for a persistent utilization spike.
  • This information can be uploaded through the management console Maintenance tab or captured from the URL https://<proxy_ip>:8082/Sysinfo

Event Log

  • The Event Log should be captured after the CPU utilization has returned to normal or after 20 minutes of high utilization for a persistent utilization spike.
  • This information can be uploaded through the management console Maintenance tab or captured from the URL https://<proxy_ip>:8082/Eventlog/Statistics

TCP Users

  • While the CPU utilization is high, copy the output from the URL https://<proxy_ip>:8082/TCP/Users

SysInfo_stats snapshots

Configure snapshots on the Edge SWG to occur every five minutes (default is 60), and run for at least 20 minutes during the CPU spike.

Full core (optional)

Depending on the nature and symptoms of the high utilization issue, you may be asked to provide a full core dump of the Edge SWG (ProxySG).

Powered by Wolken Software