Connection to database failed. Error: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.

Products

• IT Management Suite

Issue/Introduction

The NS error log is filled with errors connecting to the database during event processing and client configuration request processing. The exception type is Altiris.NS.Exceptions.DatabaseNotReadyException and the error message includes the following detail:

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may
have occurred because all pooled connections were in use and max pool size was reached

A sample of the error data generated during a client configuration request is below: 

Unable to get the client policies for specified resource (Resource: {00000000-0000-0000-0000-000000000000}, Exception: Altiris.NS.Exceptions.DatabaseNotReadyException: Failed to construct DatabaseContext object. Connection to database failed. ---> Altiris.NS.Exceptions.AeXException: Failed to open database connection. Current user is DOMAIN\USER. Error: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached. ---> System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

Cause

.NET manages a finite set of pooled connections to the SQL database and configuring NS to allow too many concurrent configuration requests can quickly use up all available connections in the pool and greatly decrease NS performance.

In one case, the customer had modified the CoreSettings.config to increase the MaxConcurrentConfigRequests value to 100 on a Notification Server, which was managing over 15,000 clients thinking that it would allow the server to process more work in a shorter amount of time. However, during busy periods of the day, all available SQL connections in the connection pool managed by .NET were used up by configuration requests (GetClientPolicy) and package info requests (GetPackageInfo) causing many other processing requests to wait for the next available connection and resulting in timeouts when connections did not become available quickly enough.

Resolution

When these errors are present regularly in the NS log, the MaxConcurrentConfigRequests value should be lowered:

If the Altiris NS Configurator is installed (included in NS 6.0 SP3, add-on to NS 6.0 SP2):

1. Open the Altiris NS Configurator.
2. Search for MaxConcurrentConfigRequests.
3. Set the value lower—10 is the recommended starting point for NS 6.0 SP3 and 20 is the recommended starting point for NS 6.0 SP2.
4. NS should immediately load the updated config on Apply.

If the Altiris NS Configurator is not installed

1. Open directory C:\Program Files\Altiris\Notification Server\Config (default install).
2. Make a backup of the existing CoreSettings.config file.
3. Carefully open the CoreSettings.config file (make a backup first!) located under a default install in and search for MaxConcurrentConfigRequest.
4. Set the Value lower—10 is the recommended starting point for NS 6.0 SP3 and 20 is the recommended starting point for NS 6.0 SP2.
5. Save the CoreSettings.config file.
6. NS should immediately load the updated config when the file is saved.

Additional Info
The CoreSettings.config value for MaxConcurrentConfigRequests should only be increased if careful monitoring of the NS and Altiris Agents shows that doing so increases performance. Increases should be done in small increments and the goal is find a balance between the maximum number of clients that can ask for a configuration request without negatively impacting all other processing.

NS 6.0 SP2 shipped with a default MaxConcurrentConfigRequests value of 50 and that proved to be too high for most large enterprise environments¹. Testing internally within Altiris and in conjunction with several customers showed that NS performs much better when MaxConcurrentConfigRequests was set to a value lower than 30. NS 6.0 SP3 took this a step further by shipped with a default value of 10 and adding a new CoreSettings.config setting name MaxConcurrentPackageInfoRequests, also with a default of 10. Prior to SP3, MaxConcurrentConfigRequests was the ceiling for both configuration requests (GetClientPolicy) and package info requests (GetPackageInfo) taken as a whole.

---

1. A large enterprise environment would be 10,000+ clients per NS, although the issue could occur in smaller environments depending on selected configuration settings. 

Applies To
NS 6.0