Unable to register Layer7 API gateway policy manager with SiteMinder policy server

Products

• SITEMINDER
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign-On
• CA Single Sign On SOA Security Manager (SiteMinder)

Issue/Introduction

Layer 7 admin is unable to register Layer7 API gateway policy manager with SiteMinder policy server.

Layer7 API gateway policy manager provides UI interface, where trusted host can be registered with SiteMinder policy server.

Required input fields are:

Address:

HostName:

Host Configuration:

FIPS Mode:

User Name:

Password:

All data values are provided, when hit "register", gets an error "Registration failed: Unable to invoke smreghost program". 

Siteminder smps.log has this error.

[SmObjKeyManagement.cpp:459][ERROR][sm-Server-03080] Failed to decrypt persistent key

 

Environment

SiteMinder Release : 12.8.06

Layer7 API Gateway version: 10.1.00-11620

Cause

Either persistent key is out of sync among different policy servers or persistent key value is somehow empty.

This can be verified by running command (smkeyexport -dadminname -wadminpw -okeyfilename -c) on each policy server, then compare output file content.

Resolution

Setting/adding AllowEmptyEncKey in sm.registry file or windows registry directly resolved the issue.

REGISTRY KEY:
HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\ObjectStore
DWORD key: AllowEmptyEncKey
Value: 1

AllowEmptyEncKey instructs Policy Server to use an empty persistent key to encrypt policy store data if Policy Server fails to decrypt the persistent key from key store.

0 - Disable
1 - Enable

In addition, when a single Policy Server generates encryption keys in an environment with multiple Policy Servers that connect to disparate policy stores, but share a central key store, an additional registry setting is required. This registry setting configures each Policy Server to poll the common key store and retrieve new encryption keys at a regular interval.

Change the following registry value: 
"EnableKeyUpdate"=0 to  "EnableKeyUpdate"=1

Restart the Policy Server.

Additional Information

Manage the Session Ticket Key

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/administrating/manage-encryption-keys/manage-the-session-ticket-key.html

How to change R12 behavior when a empty persistent key is present in the key store?