Error: Error in DSig - Can't create SMKeyDatabase in Policy Server

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On SITEMINDER CA Single Sign On Federation (SiteMinder)

Issue/Introduction

After upgrading Policy Server from 12.7 to 12.8.01, the below error messages are seen on all SAML federation partnerships.

Anytime an SP initiated authnrequest is attempted, it fails and we see the following in the FWSTrace logs and smps.log:

[Wed Nov 21 2018 09:22:41][SAMLAuthnRequestTunnelService.java][ERROR][sm-FedServer-00330]
Exception when generating AuthnRequest: com.netegrity.SAML2Security.DSigException:
Error in DSig - Can't create SMKeyDatabase.Exception occurred during creation of the XMLDocumentOps instance.
Exception: org/slf4j/LoggerFactory 
com.netegrity.smkeydatabase.api.XMLDocumentOpsException: Exception occurred during creation of the XMLDocumentOps instance.
Exception: org/slf4j/LoggerFactory 

  at com.netegrity.smkeydatabase.api.XMLDocumentOpsFactory.getXMLDocumentOpsInstance(XMLDocumentOpsFactory.java:95) 
  at com.netegrity.SAML2Security.DSigSigner.initialize(Unknown Source) 
  at com.netegrity.SAML2Security.DSigSigner.<clinit>(Unknown Source) 
  at com.netegrity.saml2ps.tunnel.SAMLAuthnRequestTunnelService.signRawXML(Unknown Source) 
  at com.netegrity.saml2ps.tunnel.SAMLAuthnRequestTunnelService.tunnel(Unknown Source) 
  at com.netegrity.policyserver.smapi.TunnelServiceContext.tunnel(TunnelServiceContext.java:245)

Environment

Policy Server 12.8SP1

Cause

There may be some missing libraries in JVMOptions.txt.

Also, if some parameters are customized, they need to be added to the JVMOptions in the new environment.

The JVMOptions.txt file contains the settings that the Policy Server uses when creating the Java virtual machine that is used to support Federation Web Services.

SAML 1.x, SAML 2.0, and WS-Federation use this file.

During a Policy Server upgrade, the existing JVMOptions.txt file is renamed to JVMOptions.txt.backup. A new JVMOptions.txt file is created (1)(2).

Below is an example of a JVMOptions.txt from a NON - WORKING environment after upgrading the Policy Server to 12.8.1:

-server
-Xbootclasspath/p:/{home_policy_server}/bin/endorsed/xercesImpl.jar;/{home_policy_server}/bin/endorsed/xml-apis.jar;/{home_policy_server}/bin/endorsed/resolver.jar;/{home_policy_server}/bin/endorsed/serializer.jar
-Xrs
-Xms128m
-Xmx256m
-DNETE_PS_ROOT=/{home_policy_server}
-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
-Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Djava.endorsed.dirs=/{home_policy_server}/bin/endorsed
-Djava.class.path=/{home_policy_server}/resources;/{home_policy_server}/config/properties;/{home_policy_server}/bin/jars/smbootstrap.jar
-Djava.util.logging.config.file=/{home_policy_server}/config/properties/logging.properties

Below is an example of a JVMOptions.txt that is from a WORKING environment.

The bootclasspath, java.class.path had missing jars.

The line for org.apache.xml.security.ignoreLineBreaks was missing.

-server
-Xbootclasspath/p:/{home_policy_server}/bin/thirdparty/stax2-api-4.0.0.jar;/{home_policy_server}/bin/thirdparty/woodstox-core-asl-4.4.1.jar;/{home_policy_server}/bin/thirdparty/wss4j-ws-security-common-2.2.0.jar;/{home_policy_server}/bin/thirdparty/wss4j-ws-security-dom-2.2.0.jar;/{home_policy_server}/bin/endorsed/xercesImpl.jar;/{home_policy_server}/bin/endorsed/xmlsec-2.1.2.jar;/{home_policy_server}/bin/endorsed/xml-apis.jar;/{home_policy_server}/bin/thirdparty/slf4j-api-1.7.25.jar;/{home_policy_server}/bin/endorsed/resolver.jar;/{home_policy_server}/bin/endorsed/serializer.jar
-Xrs
-Xms128m
-Xmx256m
-DNETE_PS_ROOT=/{home_policy_server}
-Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
-Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Djava.endorsed.dirs=/{home_policy_server}/bin/endorsed
-Djava.class.path=/{home_policy_server}/resources;/{home_policy_server}/config/properties;/{home_policy_server}/bin/jars/smbootstrap.jar;/{home_policy_server}/bin/thirdparty/log4j-api-2.10.0.jar;/{home_policy_server}/bin/thirdparty/log4j-core-2.10.0.jar;/{home_policy_server}/bin/thirdparty/log4j-slf4j-impl-2.10.0.jar
-Djava.util.logging.config.file=/{home_policy_server}/config/properties/logging.properties
-Dorg.apache.xml.security.ignoreLineBreaks=true

Resolution

Edit the JVMOptions.txt of the non-working environment to include the above highlighted:

The missing .jars;
-Dorg.apache.xml.security.ignoreLineBreaks=true;
And any customized parameters the Policy Server uses.