SP initiated saml request fails on 12.8sp2 with error in smps.log "Error Encrypting NameID."

book

Article ID: 132000

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) AXIOMATICS POLICY SERVER CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On

Issue/Introduction

SP initiated saml request fails on 12.8sp2 with error in smps.log
"Error Encrypting NameID", and
AssertionGenerator postProcess() returns fatal error.
 
User has valid session and valid certificate, correct cert alias is used for encryption.
FWStrace log will show failure like this.
[02/27/2019][13:04:14][3257][2317166336][1d6992b1-2dcc6ae4-02a80cb3-15f2d4c2-21b9c789-9bf][SSO.java][processAssertionGeneration][Transaction with ID: 1d6992b1-2dcc6ae4-02a80cb3-15f2d4c2-21b9c789-9bf failed. Reason: FAILED_INVALID_RESPONSE_RETURNED]
[02/27/2019][13:04:14][3257][2317166336][1d6992b1-2dcc6ae4-02a80cb3-15f2d4c2-21b9c789-9bf][SSO.java][processAssertionGeneration][Denying request due to "NO" returned from SAML2 assertion generator.]

Policy server trace log shows Primary certificate to verify signature: samlConfigData.metaSerialNumber: "15979cxxxxx" samlConfigData.metaIssuerName: "CN=XXXX, OU=XXXX, O= XXXX, L=XXXXXX, ST=XX, C=US"
But screen shot shows SerialNumber: 015979cxxxxx, log misses leading 0, not sure could be a cause or not at the time.

smps.log:
[31591/139937085101824][Mon Mar 04 2019 11:51:01][AssertionGenerator.java][ERROR][sm-FedServer-00130] postProcess() returns fatal error.
Error Encrypting NameID.

With FULL tracing is on, smtracedefault.log shows below:
[03/04/2019][11:51:01.395][31591][139937085101824][ProtocolBase.java][encryptNameID][Error Encrypting Assertion:0 org/codehaus/stax2/XMLOutputFactory2][][][][][][][][][][][1a519574-a8ffcbc3-506eb36f-0af6c6ad-7ba701f5-faa][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]
[03/04/2019][11:51:01.397][31591][139937085101824][AssertionGenerator.java][invoke][AssertionHandler postProcess() failed. Leaving AssertionGenerator.][][][][][][][][][][][1a519574-a8ffcbc3-506eb36f-0af6c6ad-7ba701f5-faa][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][]
[03/04/2019][11:51:01.398][31591][139937085101824][SmJavaAPI.cpp:1248][JavaActiveExpression][Active Expression evaluated for SmJavaAPI: JavaActiveExpression successfully invoked. Parameter and result follow:][NO][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][][com.netegrity.assertiongenerator.AssertionGenerator -AssertionHandler:SAML20][]

Cause

Wrong stax2 api jar was used or referenced by policy server.  stax2-api-3.1.4.jar under ~siteminder/bin/thirdparty/ is the correct dependency file to match with xmlsec-2.1.2.jar.

Environment

Policy server OS version : RHEL6 64 bit. Linux 2.6.32-642.13.1.el6.x86_64
Policy server version : Version: 12.8; Update: 02.00; Build: 1992; CR: 00;
IDP: Siteminder
SP: 3rd party
 

Resolution

12.8sp2 Policy Server is shipped with stax2-api-3.1.4.jar. 
And default 12.8sp2 JVMOptions.txt also contains entry for stax2-api-3.1.4.jar. 
However, sometime in 12.8sp1 release, policy server was shipped with both stax2-api-3.1.4.jar and stax2-api-4.0.0.jar.
JVMOptions.txt could be pointed at stax2-api-4.0.0.jar at some point if in-place upgrade was done.
 
If that is the case, JVMOptions.txt should be verified and edited to use stax2-api-3.1.4.jar.   
After the configuration change and recycling policy server, issue is solved.  DE408143