SPS server services are not starting

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• SITEMINDER

Issue/Introduction

After changing the key and certificate in apache httpd, apache httpd would not start. In error log:

[Thu MM DD 16:37:39.982349 yyyy] [ssl:emerg] [pid 1772:tid 512] AH02577: Init: SSLPassPhraseDialog builtin is not supported on Win32 (key file <install-Dir>/secure-proxy/SSL/keys/server.key)
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] AH02312: Fatal error initialising mod_ssl, exiting.
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] AH02564: Failed to configure encrypted (?) private key www.example.com:443:0, check <install-Dir>/secure-proxy/SSL/keys/server.key
[Thu MM DD 16:37:39.982349 yyyy] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=RSA)
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib
[Thu MM DD 16:37:39.982349 yyyy]] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Thu MM DD 16:37:39.982349 yyyy] [ssl:emerg] [pid 1772:tid 512] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO)
AH00016: Configuration Failed

Environment

Release : 12.8

Component : SiteMinder Federation(Federation Manager)

Resolution

Customer had modified <AccessGateway>\httpd\conf\extra\httpd-ssl.conf, setting:

SSLPassPhraseDialog builtin

Apache httpd does not support builtin on windows, and in any case, this should be set to:

SSLPassPhraseDialog custom

Set to custom and httpd would start.