SSL sessions rejected with error "OpenSSL RSA operation failure" when decrypting on SSL Visibility
Article ID: 169325
SSL Visibility Appliance Software
When using the SSL Visibility Appliance product you may notice that SSL sessions to various Google domains (and possibly others) are being rejected with the session log indicating that there is an OpenSSL RSA operation failure. This has been observed when using Chrome (56.0.2924.76) and Firefox (51.0.1), and only if the policy action is to inspect the flow.
The error is caused by a signature algorithm being used that was pulled from the TLS1.3 spec called RSA-PSS.
Release 188.8.131.52 addresses the issue and is available for download on BTO as of January 25th, 2017.
There is a mention of the new signature support in the release notes as follows: SSL Visibility 184.108.40.206 allows clients to authenticate servers using the RSA-PSS signature scheme.
If an upgrade is not feasible at this time, a workaround for the issue is to cut-through the domains being rejected.
The most efficient way to do this is by creating a custom Subject/Domain Name list for Google sites and applying it to a cut-through rule within the ruleset. The Subject/Domain Name List should contain, at the very least, the following entries: