Assertion decryption is failing

Products

• CA Single Sign On Federation (SiteMinder)

Issue/Introduction

After upgrading the Policy Server from 12.7 to 12.8.5, the Policy Server is failing to decrypt the encrypted assertions from one external federation partner:

Logs:
[520de310-59e1920c-2485d649-bbddfb97-257e6411-4][][][][11/05/2021][][][][][][717788][][][][][][][][][][][140674934499072][][][][][decryptAssertionInResponse][Exception has occured: Error in SAML2EncryptDecrypt decrypt - failed to decrypt Assertion. encrypt: Error decrypting XML Document. Error setting private key. Incorrect private key may have been used for decryption.java.security.InvalidKeyException: unwrapping failed at org.bouncycastle.jcajce.provider.BaseSingleBlockCipher.engineUnwrap(Unknown Source) at javax.crypto.Cipher.unwrap(Cipher.java:2553) at com.netegrity.smkeydatabase.api.EncryptedKeyResolver.engineCanResolve(EncryptedKeyResolver.java:123) at com.netegrity.smkeydatabase.api.XMLEncryptDecryptApacheImpl.decrypt(XMLEncryptDecryptApacheImpl.java:442)

Environment

Release : 12.8.x

Component : SITEMINDER -POLICY SERVER

Cause

12.7.x used different libraries for assertion decryption and these libraries were allowing a digest algorithm to be used that should have been rejected.

Resolution

The 12.8.5 release is working by design.  The 12.7 release used older libraries for decryption and those libraries were behaving less strictly than the newer libraries in 12.8.5.  

In 12.8.x, when key transport algorithm i.e is rsa-oaep-mgf1p (http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p) is used then the Digest Method must always be sha1.