When using the SSL Visibility Appliance product you may notice that SSL sessions to various Google domains (and possibly others) are being rejected with the session log indicating that there is an OpenSSL RSA operation failure. This has been observed when using Chrome (56.0.2924.76) and Firefox (51.0.1), and only if the policy action is to inspect the flow.

The error is caused by a signature algorithm being used that was pulled from the TLS1.3 spec called RSA-PSS.
 

Release 3.11.2.1 addresses the issue and is available for download on BTO as of January 25th, 2017.

There is a mention of the new signature support in the release notes as follows: SSL Visibility 3.11.2.1 allows clients to authenticate servers using the RSA-PSS signature scheme.

Workaround

If an upgrade is not feasible at this time, a workaround for the issue is to cut-through the domains being rejected.

The most efficient way to do this is by creating a custom Subject/Domain Name list for Google sites and applying it to a cut-through rule within the ruleset. The Subject/Domain Name List should contain, at the very least, the following entries:

CN=*google.com
CN=*googlevideo.com
CN=*googleapis.com

More entries can be added to the custom list as needed.

Instructions on how to create a Subject/Domain Names List can be found in the SSL Visibility Appliance Administration & Deployment Guide.