CA Access Gateway startup error "Pass phrase decryption failed" or "Failed to configure encrypted (?) private key"

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• SITEMINDER

Issue/Introduction

Enabling SSL/HTTPS on CA Access Gateway servers, by following these steps:

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/configuring/ca-access-gateway-configuration/configuring-ssl-for-ca-access-gateway/configuring-ssl-on-apache-web-server-manually.html

CA Access Gateway is in FIPS COMPAT Mode, a private encrypted key is generated. Server.crt and server.key are in place.
After running "configssl.bat -enable password", could not start Access gateway apache Service on Windows, however "CA Access Gateway Proxy" service started just fine.

The error log says "Failed to configure encrypted (?) private key."

apache error log:
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] Init: Unable to read pass phrase [Hint: Pass phrase decryption failed]
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=RSA)
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib
[Wed mm-dd 13:35:35.475049 yyyy ] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Wed mm-dd 13:35:35.475049 yyyy] [ssl:error] [pid 4808:tid 512] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO)
[Wed mm-dd 13:35:35.475049 yyyy] [ssl:emerg] [pid 4808:tid 512] AH02312: Fatal error initialising mod_ssl, exiting.
[Wed mm-dd 13:35:35.475049 yyyy] [ssl:error] [pid 4808:tid 512] AH02579: Init: Private key not found
[Wed mm-dd 13:35:35.475049 yyyy] [ssl:emerg] [pid 4808:tid 512] AH02312: Fatal error initialising mod_ssl, exiting.
[Wed mm-dd 13:35:35.475049 yyyy] [ssl:emerg] [pid 4808:tid 512] AH02564: Failed to configure encrypted (?) private key www.example.com:443:0, check <install-Dir>/CA/secure-proxy/httpd/conf/spsapachessl.properties
AH00016: Configuration Failed

Environment

Release : 12.8.03

Component : SITEMINDER SECURE PROXY SERVER

Cause

When server.key was generated, customer used special character inside pass phrase.
configssl.bat can not handle the special character during encryption, thus result file content of ssl.key.password within spsapachessl.properties is incorrect.

Resolution

Edit configssl.bat by add in "setlocal DisableDelayedExpansion" before the command execute. After the command, add "setlocal EnableDelayedExpansion" to have that back to original setting.

i.e.:

:encryptpwd
setlocal DisableDelayedExpansion
"%SPS_JAVA_HOME%\bin\java.exe" -classpath "%NETE_SPS_ROOT%\Tomcat\lib\proxyutils.jar;%NETE_SPS_ROOT%\Tomcat\lib\smi18n.jar;%CA_SPS_CP_RESOURCES%" com.netegrity.util.SSLUtil %2
setlocal EnableDelayedExpansion
go to done

Next, re-run "configssl.bat -enable password". This will recreate spsapachessl.properties file content.

Restart Secure Proxy Server services.

To avoid the work around altogether, do not use special character inside pass phrase when generating server.key, or not using pass phrase at all.

Additional Information

https://knowledge.broadcom.com/external/article?articleId=46380

https://knowledge.broadcom.com/external/article?articleId=202015