Error: "Connection request rejected. Connection limit of 'x' exceeded" in Policy Server

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On Agents (SiteMinder) SITEMINDER CA Single Sign-On

Issue/Introduction

The Policy Server rejects incoming connections and logs the following error to "smps.log" when it reaches its configured connection limit:

Connection request rejected. Connection limit of "x" exceeded.

For example:

Connection request rejected. Connection limit of 2048 exceeded.

This causes a service outage for the agents that cannot connect.

Environment

Policy Server, all supported versions, on Linux and Windows

Cause

An increase in load on the environment requires re-tuning the Policy Server to handle the higher request volume in a timely manner.

Several factors and Worker Thread-related tuning parameters must be evaluated to prevent Agent connections from building up under increased load.

The Policy Server communicates with User Directories, User Databases, or both to process each request. The Maximum Threads value, configured in the Policy Server Management Console (smconsole), controls how many Worker Threads are available. These Worker Threads process the IsProtected, IsAuthenticated, and IsAuthorized requests received from Agents.

Resolution

Tune the environment using the following approaches.

Policy Server side

Increase Worker Threads (default: 8), MaxConnections (default: 256), and PriorityThreadCount (default: 5) to match the increased load:

1. Stop the Policy Server.
2. Back up the registry file before making any changes.
3. On Linux, open "{home_policy_server}/registry/sm.registry". On Windows, open the equivalent key under "HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\PolicyServer" using "regedit.exe".
4. Update the following values as needed. Values prefixed with "0x" are hexadecimal. See [1] for the full field description, default value, and valid range of each parameter.
- "Thread Pool Size" — corresponds to the "Maximum Threads" field in the smconsole. Controls worker thread capacity. Default: 8. Example: "0x8".
- "Tcp Max Server Connections" — corresponds to the "Max Connections" field in the smconsole. Sets the maximum connection limit. Default: 256; this value can be increased significantly for high-load deployments. Example: "0x100".
- "PriorityThreadCount" — add this DWORD value if it does not already exist. It reserves threads for high-priority messages. Valid range: 5–20; default: 5. Any value outside this range disables the setting and reverts it to the default of 5.

On Linux, also confirm the account running the Policy Server has a "nofiles" (open file descriptor) limit well above the new "Tcp Max Server Connections" value.

Each connection, plus directory and log file handles, consumes a file descriptor, so a "ulimit -n" set too low causes socket errors instead of accepting connections. See [2] for the recommended "ulimit -n" value.

5. Save the registry file.
6. Start the Policy Server.

After restarting, run "smpolicysrv -stats" to confirm the new thread and connection values are in effect. For an explanation of this command's output, see [3].

For guidance on sizing Worker Threads before applying these values, see [4].

Agent side

Configure "MaxResourceCacheSize" [5] and "MaxSessionCacheSize" [5] so agents cache Policy Server decisions locally and send fewer requests to the Policy Server.

Bootstrap distribution

Add multiple "policyserver=" lines to each agent's "SmHost.conf" file [6]. This lets the agent's initial connection fail over to another Policy Server if the first one is unavailable.

"SmHost.conf" and a Host Configuration Object (HCO) are not the same object.

"SmHost.conf" controls only the initial bootstrap connection, while the HCO configures the agent's connections afterward [7].

Give each agent its own HCO, and list the Policy Servers in a different order in each HCO, so agent load spreads across the available Policy Servers instead of concentrating on one.

Directory load balancing

If the directory server is the bottleneck, distribute worker threads across multiple directory server entries instead of concentrating all connections on a single directory server. For additional performance tuning guidance, see [8].

As a rule of thumb, configure one User Directory Server in the initial Load-Balance Group for every 8 Worker Threads. This ratio provides enough LDAP handles so no single Worker Thread is blocked waiting for a directory connection.

For example, a Policy Server configured with 48 Worker Threads needs 6 User Directory Servers in the initial Load-Balance Group of the User Directory Definition (48 ÷ 8 = 6).

Since Release 12.8.07, the "Pool Size" setting on an LDAP user directory offers a more direct fix for this same bottleneck. Without connection pooling, the Policy Server opens only one connection set to the primary server in each LDAP bank and processes one authentication or authorization request at a time on that connection.

Raising Pool Size lets the Policy Server open multiple concurrent connections to the primary server in each LDAP bank, so it services several LDAP requests in parallel instead of queuing them behind a single connection, reducing wait time on the Policy Server side under high load.

This benefit is most noticeable when the directory sits behind a hardware load balancer or virtual IP, since that setup would otherwise reduce the directory connections to a single connection set regardless of how many backend directory servers exist.

Configure Pool Size in the Administrative UI under "Infrastructure > Directory > User Directories", on the LDAP user directory used as the user store.

The default value is 1; the maximum is 32 per directory, and Pool Size multiplied by the number of LDAP banks in a failover group must not exceed 64 (for example, a failover group with 4 banks allows a maximum Pool Size of 16 per bank). See [9] for full configuration details, and [10] for how connection pooling relates to load balancing and failover across multiple LDAP directory servers.