Avoiding CA SiteMinder Policy Server hang or crash on Microsoft Windows platforms
Article ID: 54973
Updated On:
Products
Issue/Introduction
Introduction
The SiteMinder Policy Server leverages industry-standard components and drivers to deliver common features such as encryption, database connectivity, and x5.09 certificate processing. CA maintains certification and quality assurance processes to ensure compatibility and reliability of these components. Occasionally, these components require specific environment settings or minimum patch levels for proper operation. In these situations, CA documents the requirements in the installation guides and/or product release notes.
CA SiteMinder Policy Servers that are installed on Microsoft Windows platforms require such a setting to disable a low-level threading option that is enabled by default in one of the LDAP component libraries. This document describes how Windows environment variable, NSPR_NATIVE_THREADS_ONLY, is required to disable this threading option and ensure operational stability of the CA SiteMinder Policy Server. Failure to set this environment variable properly may result in intermittent Policy Server process hang or crash failures.
CA SiteMinder Use of the Sun Directory SDK for C
CA SiteMinder Policy Servers allow LDAP directory servers to be configured as SiteMinder policy stores and SiteMinder user directories. CA SiteMinder Policy Servers use the Sun Directory SDK for C to create and maintain TCP connections and perform LDAP operations:
- For any LDAP-based policy store.
- For all SiteMinder User Directories defined with the LDAP namespace.
CA SiteMinder 6.0 Policy Servers that are installed on Microsoft Windows platforms offer the option to use Microsoft libraries for LDAP-related operations to Microsoft Active Directory or Microsoft ADAM user repositories. The Policy Servers use the Microsoft libraries only in conjunction with SiteMinder User Directories defined using the "AD" namespace. Even on Microsoft Windows, the Policy Server utilizes the Sun Directory SDK components for operations to user directories defined with the LDAP namespace or for operations to any LDAP-based policy store.
NSPR Library Threading Model
One of the Sun LDAP libraries, NSPR, provides "PRThreads" for use by the other LDAP libraries, which internally create and switch between multiple PRThreads. By default, the Microsoft Windows-compatible port of NSPR utilizes Microsoft fibers to implement PRThreads unless the environment variable NSPR_NATIVE_THREADS_ONLY is set to 1.
Unfortunately, the LDAP NSPR use of PRThreads is incompatible with recent versions of Microsoft Windows fibers. Under certain conditions, the NSPR use of Microsoft fibers may result in "thread identity switching" wherein the process stack and register values of a thread-of-control operate with a thread context different than what had been running with. This thread context switching can have unpredictable results on internal operations.
Known Problems with NSPR use of PRThreads
The thread identity switching that may result from the PRThreads use of Microsoft fibers has been shown to cause the following problems:
- COINIT errors. A thread-of-control which had had COM initialized as a SiteMinder thread pool thread then receives the thread context from an LDAP ping thread, which doesn't initialize COM. When the thread-of-control attempts to authenticate a Windows Domain user and access the Microsoft ADSI driver, the COINIT error is raised.
Example from Policy Server Authentication log:
ADsOpenObject failed on 'WinNT://PS'. ADSI Error 800401f0. CoInitialize has not been called. - Policy Server process hangs. Typically, the Policy Server process hang results when a thread that takes a resource lock and then "switches its identity" is not be able to release the lock. Similarly, a thread which has incremented a reference count and then "switched its identity" might not be able to decrement the reference count correctly so that SiteMinder shutdown cannot complete.
- Policy Server process crashes. For example, a thread identity switch may lose exception handling settings so that try-catch code would not capture a signal for an exception. The unhandled exception is then handled by the C++ runtime library, which results in a crash.
Disabling NSPR PRThreads
The Sun Directory SDK provides a facility to disable PRThreads and the NSPR library's corresponding use of Microsoft fibers. To utilize this facility and disable PRThreads, define the Windows system environment variable NSPR_NATIVE_THREADS_ONLY with the value of one (1).
NSPR_NATIVE_THREADS_ONLY=1
When the NSPR_NATIVE_THREADS_ONLY environment variable with a value of one (1) exists at Policy Server start up, the Policy Server will use only Microsoft Windows native threads.
While the environment variable NSPR_NATIVE_THREADS_ONLY=1, is a required setting for CA SiteMinder Policy Servers on Microsoft Windows platforms, the SiteMinder Policy Server installation does not create or change this environment variable. The Sun Directory SDK and its NSPR library are common components used by other vendors in their applications. These binaries are also frequently found in custom developed solutions. The NSPR_NATIVE_THREADS_ONLY=1 variable is a global system environment that will impact the behavior of all applications that use the NSPR library. To ensure systems administrators consider this impact and the compatibility of other applications on the Policy Server system, the NSPR_NATIVE_THREADS_ONLY=1 variable must be set manually.
Conclusion
Without the system environment variable NSPR_NATIVE_THREADS_ONLY set to 1, a Microsoft Windows-based CA SiteMinder Policy Server may crash or hang. Defining this environment variable disables the low-level threading option in one of the Policy Server component libraries that triggers the crash. While this environment setting is required, system administrators should review other applications on the Policy Server for potential dependencies on the the NSPR library feature and compatibility with the Policy Server.
Environment
Component: SMFSS
Feedback
