WebAgent Policy Server Connections.
Article ID: 55025
Updated On:
Products
Issue/Introduction
Description:
Policy Server ran out of connections and as a result webagent started giving 20-003 Errors.
Following Error was logged in Policy Server Smps logs-
====================================== 26/Apr/2004:17:34:50 -0800][6-Server] Connection request rejected. Connection limit of (256) exceeed. [26/Apr/2004:17:34:50 -0800][6-Server] Connection request rejected. Connection limit of (256) exceeed. =====================================
Solution:
The following TecNote below describes - How to resolve the issue of policy server running out of connections. It describes how to make sure you do not run out of sockets on the Policy Server.
Policy Server Webagent Connections
In this document, connections discussed are from the Web Agent to each Policy Server service/port rather than to the Policy Server as a whole, unless otherwise noted, as all settings are configured per port. Therefore, if no file (descriptors) is set to 1024, that means that each service has 1024 file descriptors available.
Netscape / IPlanet web servers are assumed to be configured for the default of single-process mode (MaxProcs set to 1). IPlanet v6 runs in multi-process mode by default.
Web Agents excluding Apache
The minsocketsperport and maxsocketsperport settings determine the number of sockets that at least and at most are open from the Web Agent to the Policy Server. When the web server starts (with the Web Agent enabled), it will open a minsocketsperport number of sockets. If there are multiple agent identities configured in that web server's agent configuration (in the WebAgent.conf or Web Agent Management Console), then a minsocketsperport number of sockets will open for each agent identity for 4.x Web Agents, or for each trusted host for 5.x Web Agents (see Calculations for more information).
As load increases, the number of sockets will increase as well, up to maxsocketsperport for each agent identity / trusted host. If you have more load than can be handled by maxsocketsperport, then a certain number of overflow requests will be placed in a queue (of finite size depending on the version of SiteMinder - 4.5.1 SP1 and later has a limit of 300).
Each request uses a socket, but not all requests open new sockets. If all sockets from the connection pool are in use, then additional sockets will be opened as needed, in steps of newsocketstep, until maxsocketsperport is reached. If maxsocketsperport is set to 20, this means that a maximum of 20 simultaneous requests per service can take place (e.g. 20 authentications). Sockets are not multiplexed, meaning that a socket is utilized until the response comes back from the Policy Server. Once a request is completed, the socket is placed into a connection pool so that it can be used again.
Connections should be considered persistent for all intents and purposes, meaning that once a socket is opened, it will not be closed. Exceptions include communication errors between the agent and Policy Server and the idling out of connections by the Policy Server. Socket(s) will be closed by the Policy Server if they are unused for the length of time specified by the TCP Idle Session Timeout for that service (specified in the Registry - see the Firewall Timeout Technote for more information).
Failover/Load Balancing:
If failover or load-balanced Policy Servers is configured, then the Web Agent will open minsocketsperport sockets to each upon startup. Connections to a load-balanced Policy Server occur in the same way as to a single Policy Server, although fewer sockets may be opened to each Policy Server as each is getting half of the total requests. If failover is configured and an error occurs between the Web Agent and the primary Policy Server that causes failover to occur, then connections to the failover Policy Server will be used and will be handled as described above. Failover occurs per service, so there may be active connections to both the primary and the failover Policy Servers at once. Once the primary Policy Server comes back up, the sockets opened to the failover server remain. All new sockets will be opened to the primary Policy Server. Failover is inherently part of load balancing; if one of the load balancing Policy Servers becomes unavailable, normal failover will take place.
Apache Web Agents
Unlike the other agents, Apache Web Agents do not use connection pooling. Apache is multi-processed and has a drastically different architecture from IIS and Netscape, which are multi-threaded. Apache spawns child processes to handle requests, and uses a configuration setting called MaxClients to determine the maximum number of child processes that it will fork to handle load. The number of child processes is managed by Apache settings in the Httpd.conf file. Each child process has its own independent socket connection(s) to the Policy Server. When the Apache parent process forks a child, an initial connection is opened to each policy server for the default agent. For 4.x Web Agents, because each socket is specific to a particular agent identity, each additional agent identity defined requires its own socket per child. A socket is opened the first time the child handles a request for an agent identity. This means that each child can have as many open sockets as there are agent identities (including the default agent identity). As load is evenly distributed among the children by the Apache parent process, the total number of sockets opened from an Apache server at maximum will equal MaxClients times the number of agent identities. For 5.x Web Agents, the total number of sockets opened from an Apache server at maximum will equal MaxClients times the number of trusted hosts.
This connection model may have major implications for the Web Agent to Policy Server ratio (depending on the version of the Policy server being used), as the limiting factor often becomes connections between the agent and Policy Server rather than the number of transactions per second. Before deploying SiteMinder on Apache, it is very important to ensure that the Policy Server is able to handle the maximum number of connections that may be opened by all Web Agents that connect to that Policy Server. Please see the Calculations section below for additional information.
Failover/Load Balancing:
The Apache agent opens the same number of connections to all configured Policy Servers, whether or not failover has occurred. Since each child process has its own connection(s) to the Policy Server, failover occurs independently for each child. This can result in a 500 error for each socket as failover is taking place. Once the primary Policy Server comes back up, the sockets opened to the failover server remain. All new sockets will be opened to the primary Policy Server. Failover is inherently part of load balancing; if one of the load balancing Policy Servers becomes unavailable, normal failover will take place.
Policy Server
SiteMinder Policy Server 4.51, 4.61 pre SP4, or 5.0 pre SP1
A proper nofiles(descriptors) setting is required on Solaris Policy Servers to accommodate the sockets being opened by the Web Agents. This is done by running ulimit -n 1024 (sets nofiles file descriptors to 1024), which is usually placed in smuser's .profile file so that it runs whenever smuser logs in (su - smuser). Therefore, you can have 1024 total sockets/files (a file descriptor is used for each) per process. This is for all sockets and files used by the process, which may include, besides connections to the agent, connections to such objects as the user directory and log files.
The Policy Server is set by default to allow 256 Web Agent connections (MaxSockets in smconsole or the Policy Server Management Console). The Solaris Policy Server has a known limitation that once a total of 1024 sockets are in use, the policy server will no longer be able to receive connections resulting in handshake errors. Therefore, MaxSockets should be increased to no more than 950 (to leave room for sockets to objects other than the Web Agent). Even if nofiles(descriptors) is increased to more than 1024, MaxSockets should not be increased past this limit.
SiteMinder Policy Server 4.61 SP4, 5.0 SP1, or 5.5, or later
A proper nofiles(descriptors) setting is required on Solaris Policy Servers to accommodate the sockets being opened by the Web Agents. This configures the ulimit or the number of file descriptors available to each Policy Server service. The ulimit should be set to at least 1024, and may be higher depending on the system needs and the version of Solaris being used. To set nofiles(descriptors) to 1024, for example, run ulimit -n 1024; this command usually is placed in smuser's .profile file so that it runs whenever smuser logs in (su - smuser). The nofiles(descriptors) value determines the maximum number of sockets and files which may be used at the same time by the process, which may include, besides connections to the agent, connections to such objects as the user directory and log files.
The Policy Server is set by default to allow 256 Web Agent connections (MaxSockets in smconsole or the Policy Server Management Console). Due to a new socket model*** introduced in Siteminder 4.61 SP4, SiteMinder 5.0 SP1, and SiteMinder 5.5, there no longer is an internal restriction on the maximum value for MaxSockets. Therefore, the MaxSockets value may be increased up to just below the ulimit. There is a theoretical MaxSockets maximum of 32K, although Netegrity recommends setting MaxSockets no higher than 10K, the maximum tested value. Please note that some room still needs to be left when setting MaxSockets - for example, if it is calculated that there could be up to 1024 Web Agent connections, you should set MaxSockets to a slightly higher value, such as 1256.
Calculations
4.x Web Agents excluding Apache:
If there is one Web Agent (with a default agent identity only) connecting to the Policy Server, and the maxsocketsperport setting is 20, then there will be a maximum of 20 * 1 = 20 open sockets. However, if there are 20 Web Agent identities with a maxsocketsperport to 20, then there will be a maximum of 20 * 20 = 400 open sockets. Therefore, the number of sockets being opened to the Policy Server can increase very quickly if you increase the number of Web Agent identities (or maxsocketsperport). If there are any Web Agent identities using the Policy Server for failover, then minsocketsperport for each of those identities must also be added (except for Apache - see Apache agent calculations). You also should calculate the total number of sockets that will be needed on the Policy Server if all agent identities failover completely.
By default, the maximum number of agent connections allowed by the Policy Server is 256 (set in smconsole or the Policy Server Management Console under the Settings tab). This means that in the second scenario above, you have the possibility of opening more sockets on the agent side than the Policy Server can accept, causing connections to be refused. If this occurs, then with debug tracing enabled on the Policy Server, a message "Rejected connection request. Too many server threads (256) or server is shutting down." will appear in the service's debug logs, and 500 errors will appear in the browser.
4. x Apache Web Agents:
In Apache, the number of connections is calculated as one connection per Apache child process, per Web Agent identity. For example, if you have a maximum of 150 child processes (value of MaxClients in Httpd.conf) and 3 agent identities, then there will be a maximum of 150 * 3 = 450 connections from that agent. The maximum number of child processes (Apache agents) / minsocketsperport (other agents) for other Web Agent identities using the Policy Server for failover must also be added to that total.
By default, the maximum number of agent connections is 256 (set by MaxSockets in smconsole). Therefore, in this scenario, the number of client connections could exceed the number that the Policy Server can accept, causing connections to be refused. If this occurs, then with debug tracing enabled on the policy server, a message "Rejected connection request. Too many server threads (256) or server is shutting down." will appear in the service's debug logs, and 500 errors will appear in the browser.
5.x Web Agents excluding Apache:
If there is one Web Agent, and thus one trusted host, connecting to the Policy Server, and the maxsocketsperport setting is 20, then there will be a maximum of 20*1 = 20 open sockets. Even if multiple agent identities are created within that Web Agent, as long as there is only one smhost.conf file, only one set of sockets will be opened to the Policy Server. If there are any Web Agents using the Policy Server for failover, then minsocketsperport per each of those trusted hosts must also be added (except for Apache - see Apache agent calculations). You also should calculate the total number of sockets that will be needed on the Policy Server if all of the agents failover completely - see the 4.x Web Agents calculation section if not all of the agents are 5.x.
By default, the maximum number of agent connections is 256 (set by MaxSockets in smconsole). If the number of client connections exceeds the number that the Policy Server can accept, it will cause connections to be refused. If this occurs, then with debug tracing enabled on the policy server, a message "Rejected connection request. Too many server threads (256) or server is shutting down." will appear in the service's debug logs, and 500 errors will appear in the browser.
5.x Apache Web Agents:
In Apache, the number of connections is calculated as one connection per Apache child process, per trusted host. For example, if you have a maximum of 150 child processes (value of MaxClients in httpd.conf) and 1 trusted host, then there will be a maximum of 150 * 1 = 150 connections from that agent. The maximum number of child processes (Apache agents) / minsocketsperport (other agents) for other Web Agents using the Policy Server for failover must also be added to that total.
By default, the maximum number of agent connections is 256 (set by MaxSockets in smconsole). If the number of client connections exceeds the number that the Policy Server can accept, it will cause connections to be refused. If this occurs, then with debug tracing enabled on the policy server, a message "Rejected connection request. Too many server threads (256) or server is shutting down." will appear in the service's debug logs, and 500 errors will appear in the browser.
Recommendations
Web Agents excluding Apache:
For non-Apache Web Agents, if all sockets in the connection pool are being utilized, then this usually indicates that there is a bottleneck in the backend (Policy Server, user directory, etc). For that reason, and to limit the number of connections to the Policy Server, Netegrity recommends against increasing maxsocketsperport above the default of 20. With the default maxsocketsperport (Web Agent) and MaxSockets (Policy Server) settings, up to 10-15 agent identities may connect to a single Policy Server. You must make sure that the maximum number of sockets that can be opened does not exceed the capacity of the Policy Server to accept those connections, as described in the Calculations section. If using Siteminder 4.51, SiteMinder 4.61 pre SP4 or SiteMinder 5.0 pre SP1, the total number of connections to the Policy Server from all agent identities (4.x Web Agents) or trusted hosts (5.x Web Agents) should not exceed the maximum MaxSockets setting of 950.
Apache Web Agents:
For Apache Web Agents, the ratio of Web Agents to Policy Servers is more on the order of 2-4 agent identities per Policy Server, depending on the MaxSockets setting on the Policy Server and the MaxClients setting on each Apache instance and the number of agent identities. You must make sure that the maximum number of sockets that can be opened does not exceed the capacity of the Policy Server to accept those connections, as described in the Calculations section. If using SiteMinder 4.51, SiteMinder 4.61 pre SP4 or SiteMinder 5.0 pre SP1, the total number of connections to the Policy Server from all agent identities (4.x Web Agents) or trusted hosts (5.x Web Agents) should not exceed the maximum MaxSockets setting of 950.
Description of the new threading model in SiteMinder Policy Server 4.61 SP4, 5.0 SP1, and 5.5
In SiteMinder 4.61 SP4, 5.0 SP1, and 5.5, a new threading model was implemented in order to improve the scalability of the SiteMinder Policy Server. In previous versions, each new socket being opened by the Web Agent was handled by its own thread. This meant that with large numbers of Web Agents, or just a few Apache agents due to their different connection method, the large number of threads could result in context switching and overall degradation of performance as the number of active threads increased.
In the new threading model, a pool of worker threads is utilized to handle requests from the Web Agent. A reactor thread handles the initial connection from all incoming requests and passes them into a queue serviced by the worker threads. The position in the queue can differ depending on the type of incoming request. The number of worker threads in the pool is handled internally and increases and decreases depending on load. The default maximum number of worker threads is 10, which can be increased or decreased in the registry or configuration files for the Policy Server. If the maximum number of worker threads has been reached and all are in used, up to two more worker threads may be added if handshake requests need to be serviced (and stay in the pool for general requests afterwards). Varying the size of the worker thread pool is allowed, but no general guidelines exist as to values since that is dependent on the specific environment in use. For optimum results, experimentation with both higher and lower values may be beneficial.
This new threading model means that the previous limitation of 950 agent connections per Policy Server no longer applies. The limit on the number of agent connections now is based on the number of file descriptors allowed for the Policy Server services (ulimit). Netegrity has tested with up to 10K sockets. Although this means that more agents can talk to a single Policy Server than before, care should be taken not to overload the Policy Server box in general. Also, the MaxSockets value still should be somewhat less than the ulimit setting (e.g. no more than 950 with a ulimit of 1024).
Environment
Component:
Feedback
