Error 81 during search: Can't contact LDAP server in Policy Server

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On SITEMINDER CA Single Sign On Agents (SiteMinder) CA Single Sign On Federation (SiteMinder)

Issue/Introduction

The Policy Server logs the following error in `smps.log` when it searches an LDAP user store:

...
Error# '81' during search: 'error: Can't contact LDAP server' Search Query = '(uid=<userid>)' for server '<ldap_host>:<port>'
...

No problem is reported by end users when the error is logged.
A connection idle timeout may have been configured on the LDAP server, for example the CA Directory `user-idle-time` property, which is sometimes set to resolve unrelated problems such as Identity Manager connection issues.

Environment

Broadcom SiteMinder Policy Server, all versions
LDAP user store (for example Symantec/CA Directory), optionally behind a hardware load balancer or virtual IP (VIP)
Example where the issue was observed: 4 Policy Servers 12.8 SP8 on Linux, connected to 4 CA Directory user stores behind an L4 load balancer, with several user store LDAP instances behind one VIP

Cause

The LDAP server (or a network device in the path) closed the connection on its side, but the Policy Server did not close it, so the Policy Server still treats the connection as open. A CA Directory DSA disconnects a user that stays idle longer than the `user-idle-time` value [1]. Stateful network devices such as load balancers and firewalls can also end idle connections without notifying either end [2].

When the Policy Server next uses that connection, the search fails with LDAP error 81, which is logged in `smps.log`. The Policy Server trace log (`smtracedefault.log`, with the relevant LDAP trace components enabled) then shows the Policy Server recovering in the following sequence:

...
Error# '81' during search: 'error: Can't contact LDAP server' Search Query = '(...)' for server '<ldap_host>:<port>'
Reconnect to server '<ldap_host>:<port>' as it's previous connections are closed and it is available for connecting now
Ldap Search callout succeeds. (Search) Base: '<base_dn>', Filter: '(uid=<userid>)'. Status: 1 entries.
...

The `Reconnect to server` message is how the Policy Server reports that it detected the closed connection and is re-establishing it [3]. It is seen in the Policy Server trace log (`smtracedefault.log`). The Policy Server closes the broken connection, opens a new one, and completes the Agent request successfully. This is the Policy Server working as designed and does not by itself indicate a problem.

Resolution

Locate the `Error# '81' during search: 'error: Can't contact LDAP server'` entry in `smps.log`, then check the Policy Server trace log (`smtracedefault.log`) around the same time. If a `Reconnect to server` message and a successful `Ldap Search callout succeeds` message follow, and no users are affected, no action is required [3].
To stop the error from being logged, configure the LDAP user store so it does not close its connections to the Policy Server. For CA Directory, increase `user-idle-time` in the DSA configuration, or set it to `0` for unlimited [1].
If a load balancer or VIP sits between the Policy Server and the LDAP servers, ensure its idle timeout is not shorter than the LDAP idle timeout, so it does not end the connections first [2].
If the error is not followed by a successful reconnect, or users are affected, investigate LDAP server availability and response time. See [3] for a case where the LDAP server did not respond within the `LDAPPingTimeout` period.