Connection to database failed. Error: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.

Products

• IT Management Suite

Issue/Introduction

The NS error log is filled with errors connecting to the database during event processing and client configuration request processing. The exception type is Altiris.NS.Exceptions.DatabaseNotReadyException and the error message includes the following detail:

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may
have occurred because all pooled connections were in use and max pool size was reached

A sample of the error data generated during a client configuration request is below: 

Unable to get the client policies for specified resource (Resource: {00000000-0000-0000-0000-000000000000}, Exception: Altiris.NS.Exceptions.DatabaseNotReadyException: Failed to construct DatabaseContext object. Connection to database failed. ---> Altiris.NS.Exceptions.AeXException: Failed to open database connection. Current user is DOMAIN\USER. Error: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached. ---> System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

Cause

.NET manages a finite set of pooled connections to the SQL database and configuring NS to allow too many concurrent configuration requests can quickly use up all available connections in the pool and greatly decrease NS performance.

In one case, the customer had modified the CoreSettings.config to increase the MaxConcurrentConfigRequests value to 100 on a Notification Server, which was managing over 15,000 clients thinking that it would allow the server to process more work in a shorter amount of time. However, during busy periods of the day, all available SQL connections in the connection pool managed by .NET were used up by configuration requests (GetClientPolicy) and package info requests (GetPackageInfo) causing many other processing requests to wait for the next available connection and resulting in timeouts when connections did not become available quickly enough.

Resolution

When these errors are present regularly in the NS log, the MaxConcurrentConfigRequests value should be lowered:

1. Search for MaxConcurrentConfigRequests (under Settings>Notification Server>Core Settings)
2. Set the value lower—10 is the recommended starting point.
3. NS should immediately load the updated config on Apply.

Note:

You can find the same coresetting by:
Open directory C:\Program Files\Altiris\Notification Server\Config (default install).

• Make a backup of the existing CoreSettings.config file.
• Carefully open the CoreSettings.config file (make a backup first!) located under a default install in and search for MaxConcurrentConfigRequest.
• Set the Value lower—10 is the recommended starting point.
• Save the CoreSettings.config file.
• NS should immediately load the updated config when the file is saved.

 

Additional Info
The CoreSettings.config value for MaxConcurrentConfigRequests should only be increased if careful monitoring of the NS and Altiris Agents shows that doing so increases performance. Increases should be done in small increments and the goal is find a balance between the maximum number of clients that can ask for a configuration request without negatively impacting all other processing.