When using the SSL Visibility Appliance product you may notice that SSL sessions to various Google domains (and possibly others) are being rejected with the session log indicating that there is an OpenSSL RSA operation failure. This has been observed when using Chrome (56.0.2924.76) and Firefox (51.0.1), and only if the policy action is to inspect the flow.
The error is caused by a signature algorithm being used that was pulled from the TLS1.3 spec called RSA-PSS.
Release 184.108.40.206 addresses the issue and is available for download on BTO as of January 25th, 2017.
There is a mention of the new signature support in the release notes as follows: SSL Visibility 220.127.116.11 allows clients to authenticate servers using the RSA-PSS signature scheme.