Admin UI causes SiteMinder Policy Server to hang after popup error "smpolicysrv.exe failed to start because xerces-c_1_5_1.dll was not found. Reinstalling the application may fix the problem".

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

Description:

We just upgraded our policy servers to 6.0 SP5,the option pack is not installed on any of these policy servers.

Since the upgrade, we have received the following popup error on a few of the policy servers:

"smpolicysrv.exe failed to start because xerces-c_1_5_1.dll was not found. Reinstalling the application may fix the problem".

At that point, you cannot get into the Admin GUI, and the smps.log is flooded with handshake failures from agents due to socket, restarting the policy server cleared the problem.

Solution:

The problem is because the policy server is trying to load the library "xerces-c_1_5_1.dll" which is part of the SiteMinder Option Pack, causing the policy server to hang.

The issue is related to a network failure between the Policy Server and the Policy Store and it has been fixed in SiteMinder 6 SP5 CR23:

11/20/2008 6.0 SP5 CR023 contains fixes for the following tracking numbers:

Tracking # Problem description---------- -------------------73454      The SiteMinder Policy server on the windows platform will no longer display an error box on the system console 
           when a loadlibrary error occurs. This was causing the Policy Server to hang.

More infomation about Fix 73454:

<- Begin

Detail:
-------
Policy server gets into LOCK/HANG state after database shutdown/restart (Pstore) then access AdminUI

Customer has reproduce outside of production. During this customer was able to generate dumps.

Debug information show policy server is LOCKED during Admin UI - SMUTILITIES is attempting to LOAD DLL form our thirdparty software xerces-c_1_5_1.dll

Resolution:
-----------
Analysis reveals that the issue appears when the machine hosting the policy store goes out of network and comes back again. While this policy store machine is coming back to network if we try to login to Admin-UI we encounter the error "The application has failed to start because 'xerces-c_1_5_1.dll' was not found. Re-installing the application may fix this problem".

This issue appears only while trying to login to Admin-UI when the policy store machine is coming back to network.

Observations before fix:
------------------------

1. After 2-3 minutes, the error "The application has failed to start because 'xerces-c_1_5_1.dll' was not found. Re-installing the application may fix this problem" pops up. Do not "OK" it.
   
   
2. Using SmTest tool, make 8-10 requests in sequence Connect->IsProtected->IsAuthenticated->IsAuthorized. SmTest tool hangs. Policy Server hangs.
   
   
3. Clear xerces error by clicking "OK", policy server and SmTest tool will start responding again.

Observations after fix:
-----------------------

1. No error appears such as "The application has failed to start because 'xerces-c_1_5_1.dll' was not found. Re-installing the application may fix this problem".
   
   
2. Using SmTest tool, make 8-10 requests in sequence Connect->IsProtected->IsAuthenticated->IsAuthorized. No hang is seen in SmTest tool or Policy Server.

<- End

Environment

Release:
Component: SMPLC