12.8.06 certutil and SSL connections fail with FIPS_ENABLED on RHEL7

Products

• SITEMINDER

Issue/Introduction

After upgrading to 12.8.06a (12.80.0600.2658) SSL connections fail with the following error:

[SmDsLdapConnMgr.cpp:729][ERROR][sm-Ldap-01320] (SmDsLdapConnMgr(Bind): SSL client init failed in LDAP Initialization).

And certutil operations failed with:

certutil: function failed: SEC_ERROR_PKCS11_DEVICE_ERROR: A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot.

OS certutil works as expected. Issue appears to be related to FIPS Mode being turned on:

% cat /proc/sys/crypto/fips_enabled
1

With FIPS mode turned off, everything works as expected.

Environment

Release : 12.8.6 and 12.8.6a

Component : SITEMINDER -POLICY SERVER

Cause

This is a defect that was introduced in 12.8.6 and is also present in 12.8.6a.

Resolution

This defect will be corrected in the 12.8.7 release of the Policy Server.  For the 12.8.6 and 12.8.6a releases, the problem is resolved by replacing the contents of the pkcs11.txt file with the following:

library=/opt/CA/siteminder/lib/libsoftokn3.so
name=NSS Internal PKCS #11 Module
parameters=configdir='sql:/opt/CA/siteminder/certCA'  certPrefix='' keyPrefix='' secmod='secmod.db' flags= updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription=''
NSS=Flags=internal,moduleDB trustOrder=75 cipherOrder=100 slotParams=(1={slotFlags=[RSA,DSA,DH,RC2,RC4,DES,RANDOM,SHA1,MD5,MD2,SSL,TLS,AES,Camellia,SEED,SHA256,SHA512] askpw=any timeout=30})

======================================================================================================
The dashed line above is only to delineate the pkcs11.txt file contents from the rest of this article.  Do NOT include the dashed line above this one in the file contents.

Please note the example above is for a Unix Policy Server.  You may need to update the values of both the 'library' and 'configdir' parameters to match the installation path of your Policy Server.  Keep the 'sql:' prefix in front of the 'configdir' value.  For Windows Policy Servers, use double backslashes in the paths as follows:
c:\\Program Files\\CA\\siteminder\\bin\\certdb

Additional Information

Associated defect number:
DE522670