CPU Monitor reporting high CPU utilization in SSL and Cryptography

book

Article ID: 168508

calendar_today

Updated On:

Products

Advanced Secure Gateway Software - ASG Symantec WebFilter (formerly Blue Coat WebFilter - BCWF) Secure Web Gateway Virtual Appliance ProxySG Software - SGOS

Issue/Introduction

CPU Monitor for ProxySG or Advanced Secure Gateway (ASG) is reporting high CPU utilization in SSL and Cryptography.

Resolution

High CPU being reported on the ProxySG/ASG can be due to one or more of the reasons which are listed below, or something else entirely. However, the most common reasons for high CPU in SSL and Cryptography are as follows:

High SSL traffic to specific sites

To determine if there is something in the network causing continuous policy evaluations, Blue Coat Reporter can be leveraged. The following Reporter reports can be helpful to further diagnose the issue:

  • SSL access logs show protocol of TCP for SSL traffic i.e. tunneled. The SSL Access Log contains protocols of type TCP and SSL. The SSL protocol has been handed off to the SSL Proxy. 
  • Web requests per client IP report filter further based on the site and protocol. 

Additionally, the Blocked web browsing per user and Default Bandwidth reports are also useful. 

If after reviewing the access logs significant ssl traffic to specific sites is seen, apply a policy to disable SSL interception for a particular site(s). 
For example, to disable SSL interception for Facebook, Google and Microsoft apply the following policy (for explicit deployment):

<proxy>
condition=SSL_Disabled_Domains detect_protocol(none)
 
define url.domain condition SSL_Disabled_Domains
facebook.com 
google.com
microsoft.com
end

 

Browser retries after policy denial

If the reports in Blue Coat Reporter show that many HTTPS requests are the result of policy denial, then rather than denying the request use bandwidth management instead to make the site unusable. Symptoms include client request is denied but browser ignores that and retries the request for the same object.

For example, limit the server inbound to 1 Kbps. This setting slows down the performance of the site rather than completely disabling it by denying the request.

For more information on how to limit bandwidth on the ProxySG please reference the Bandwidth Management: Limiting Bandwidth Technical Brief. 

 

High number of requests from one or more clients

High CPU utilization can result when one or more clients send a high number of requests to the proxy. Examples include opening many simultaneous sessions to the same destination, or applications that do not understand proxy authentication leading to a loop of request or authentication prompts.


To identify this situation, use the management console. Navigate to Statistics > Sessions > Active Sessions and see if a single user tries to make a large number of connections to the same destination. One possible resolution for this situation is to enable the Attack Detection feature by issuing the following commands at the CLI:

proxy>enable
proxy#conf t
proxy#(config)attack-detection
proxy#(config attack-detection)client
proxy#(config client)enable-limits

 

These commands limit the number of sessions that are permitted from any client up to 100 simultaneous sessions and subsequent sessions for this client IP are dropped. Care should be used when implementing this policy as it may prevent legitimate applications from connecting correctly to the Internet.

Note: This configuration is not recommended to be done globally on the proxy in case of:

  1. This proxy is used as upstream proxy for other proxies
  2. Group of clients connect to the proxy using the same IP as they come from any Network Address Translation (NAT) device.

 

Multiple certificates using the same Common Name

Beginning in SGOS 6.5.9.9 and 6.6.4.3 a fix was introduced to address this issue.  

In SGOS releases before 6.5.9.9 and 6.6.4.3 an issue can occur where multiple certificates use the same common name. This situation causes the ProxySG to continuously recreate its version of the certificates which can be a CPU intensive process. Only if you are unable to upgrade to the latest version, the following workaround can be used. Change the SSL forward proxy policy in the VPM:

  1. Access the SSL Intercept layer.
  2. Edit the rule that is set to intercept SSL (right-click and select Edit where action contains Enable HTTPS Interception Object).
  3. Select Splash Text and enter the following in the text box below it: $(x-rs-certificate-serial-number)$(x-rs-certificate-valid-from)$(x-rs-certificate-valid-to)
  4. Click OK > Install Policy.
If the appliance has the SSL interception enabled in CPL format only (instead of VPM), change the intercept line by adding the splash text as shown in this example:
<ssl-intercept>
ssl.forward_proxy(https) ssl.forward_proxy.splash_text("$(x-rs-certificate-serial-number)$(x-rs-certificate-valid-from)$(x-rs-certificate-valid-to)")

You can monitor the cert cache by accessing the following URL on the ProxySG appliance: https://<PROXYIP>:8082/sslproxy/certcache

 

Increase the Certificate Cache

To get the most out of the changes if the proxy is running SGOS 6.5.5.1 or later, increase the certificate cache timeout from the default of 2 hours to 72 hours, from the CLI:

proxy>enable
proxy#conf t
proxy#
(config)ssl
proxy#
(config ssl)proxy set-cert-cache-timeout 72

 

Lower the emulated certificate key size

With SSL-Interception, the ProxySG emulates on the fly the certificate copies of the original certificates from the websites.

Decrease the size of the ssl-interception private keys from the CLI:

proxy>enable
proxy#conf t
proxy#
(config)ssl
proxy#
(config ssl)proxy force-emulated-cert-keysize 2048

 
 

DHE cipher introduced in SGOS 6.5

Only for SGOS 6.5 releases before 6.5.6.1, the DHE cipher was used as the default cipher and may lead to high CPU utilization.

Beginning in SGOS 6.5.6.1, DHE is disabled by default, the ECDHE cipher is added which provides a similar level of encryption with fewer CPU resource requirements. For more details on the introduction of ECDHE, please refer to ECDHE cipher support on ProxySG.

proxy>enable
proxy#conf t
proxy#
(config)ssl
proxy#
(config ssl) proxy dhe-ciphers disable

 

Untrusted Certificate on the Upstream

On a Proxy Sysinfo or Snapshot, you may see something similar to the following lines flooding the file:

2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: instance-2"  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746
2019-02-22 22:46:51-00:00UTC  "Server certificate validation failed: CERT_UNTRUSTED_ISSUER, Name in certificate: "  0 300000:1  te_transaction.cpp:1746

This is an indicative that the proxy is not trusting on a certificate from the upstream devices (possible reasons include the certificate being expired or having a common-name mismatch, amongst other possible causes).

Workaround:

- Apply the following CPL:

<SSL>
server.certificate.validate(no)             
 
- Create the following rule on the VPM, as shown in the article TECH241967:
  1. Create a Web Access Layer.
  2. Right-click Action > Set > New > Set Server Certificate Validation.
  3. Check "Disable Server Certificate Validation". Click OK and "Install Policy".

You should see the CPU utilisation lowers down drastically within a few minutes.

Note:

Disabling the certificate validation makes the ProxySG vulnerable to Man In the Middle attacks.

Solution:

In order to fix this issue permanently, it is necessary to identify the upstream devices causing the problem and troubleshoot the certificate issues presented on said devices.

To help during the troubleshooting process, reproduce the issue on a maintenance window (or before applying the fix) and:

  1. Take a Packet Capture
  2. Export the Active and Errored Sessions (Statistics > Sessions).
  3. Export the SSL Access Log.

Once you have identified the destination(s) of the errors, you may apply the rule above to only those destinations, while the problematic certificates are being troubleshot and fixed.

 

SSL Traffic Profile

Web Ads/Analytics traffic profile has little/no session reuse due to one-time ads being serviced. Low session reuse contributes high CPU.

Apply the Best Practice: SSL Decryption Policy, please refer to article Symantec Web Gateway (SWG) - Best Practice: SSL Decryption Policy to apply rules to Web Ads/Analytics category if the traffic becomes the Top Categories by Requests.