Package Servers are unable to download new or updated packages

Products

• IT Management Suite

Issue/Introduction

Package Servers are unable to download any new or updated packages. The local Altiris Agent log has the following severity 1 error:  "Error while downloading package: Server is busy. Package sources request is backing off (-2147467259)."

On the NS, the following information was found:

• Opened Task Manager and noted that the server does not appear to be working hard on any processes.
• Checked the event queues and saw that all is clear.
• Checked the IIS logs and saw that there weren't any obvious errors.
• Checked the HTTP Error log and found many server busy (status 503) errors for GetClientPolicies, "Connection_Dropped" errors for both GetClientPolicies and PostEvent and "BadConnection" errors for PostEvent

Based on the IIS and HTTPError logs, it looks like connections are being made, and then are being closed for some reason.

Cause

There are over 15,000 clients with a 4-hour config request interval, which under normal conditions should allow the NS to properly handle configuration requests and package info requests with MaxConcurrentConfigRequests value of much less than the default of 50 in NS 6.0 SP2 and versions prior. However, in this case, the customer has been using custom VB script to force configuration requests and, over a period of time, that caused most of the requests to come into the NS over a 90-minute interval rather than evenly over 4 hours.  

Also, bandwidth throttling had been recently enabled in this customer environment and this behavior had not occurred prior to that change. While bandwidth throttling provides the benefit of lower network utilization, it does create longer client connection times with the NS because clients are sending and receiving smaller amounts of data over a longer period of time.

The end result was that during specific times of the day, the combination of client config requests and package info requests were generating more than 50 concurrent requests, and Package Servers and clients were being turned away from the NS and forced to retry later.

Resolution

To see the error from the NS perspective, enable trace logging (severity=15) and enable logging for GetPackageInfo requests by updating HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\eXpress\Notification Server\LogPackageInfoEvents to a value of 1.

( LogPackageInfoEvents is a DWORD value)

The NS will log a severity 8 error that describes the problem and the fix:

The maximum number of package info requests has been reached. Tune the core setting "MaxConcurrentConfigRequests" for optimal performance. (Max: 50)

As a short-term fix to allow the Package Servers to download new or updated packages, Open the CoreSettings.Config file located in \Program Files\Altiris\Notification Server\Config folder and temporarily increase the the defaultValue for MaxConcurrentConfigRequests to a higher value than the default of 50. After the Package Servers complete their downloads, reduce this value back to the default of 50 or below.

Important: Altiris does not recommend running with this value set higher than 30 (and typically much lower is better) in NS 6.0 SP2 unless specific performance monitoring shows improvement at higher values. Running for a long time with too high of a MaxConcurrentConfigRequests value can have unintended consequences on the NS and could result in database deadlocks and database connection failures.

Note: In NS 6.0 SP3, the default value for MaxConcurrentConfigRequests has been reduced to 10 and there is a new setting called MaxConcurrentPackageInfoRequests, also with a default value of 10, which is used to limit the number of GetPackageInfo requests separately from GetClientPolicies requests. This modification separates the two limits and allows for more granular modifications for performance tuning. Again, increases to these defaults should only be made where specific performance monitoring shows an overall improvement with an increase.

Applies To
NS 6.0 SP2 with Hotfix 19
More than 15,000 clients running configuration requests every 4 hours and all set to use bandwidth throttling.
Seven Package Servers with 300 or more packages.