Isolating the Inventory Rule Management Web services

Article Id: 180378
Status: Published
Updated On: 14-01-2013 11:36
Legacy Id: HOWTO4592
Products:
Patch Management Solution for Windows
Issue/Introduction:

 

Resolution:

Problem
 

The Notification Server console performance is sluggish during the same day an update to the PMImport.cab file is released. Performance usually returns to expected levels 4 to 24 hours later. 

In extreme circumstances, the IIS server will return 503 error codes (server too busy). The 503 codes can be seen in the IIS logs, and are a symptom of exceeding the server-wide kernel request queue limit for all application pools. The majority of all IIS traffic will resolve to AgentRuleData.ashx.

Environment
 

  • Notification Servers managing large amounts of computers (greater than 5,000).
  • New PMImport.cab released in the past 24 hours.
  • Most likely to be observed when the customer also has multiple languages enabled for Patch Management Solution for Windows.
And at least one of the following solutions are installed:

 

  • Patch Management Solution for Windows 6.1, 6.2
  • Patch Management Solution for Dell 6.1, 6.2
  • Patch Management Solution for Windows 7.0 (***see note below)
  • Patch Management Solution for Windows 7.1+ (***see note below)

Cause
 

The Inventory Rule web services are not throttled. 

After a new PMImport.cab is released, the managed computers perform their regularly scheduled inventory rule processing (defaults to 4 hours). Because new rules exist, a temporary large demand spike occurs as each managed computer updates their inventory rule cache, performs the inventory scan, and then uploads the new scanning data.

Resolution
 

Disable web.config Debug compilation modeThis setting affects many Altiris Solutions, but will have a larger performance impact in regards to Patch Management functions
See article 33499 for how to disable debug mode in both of the web.config files located at:
 

 

  • \Program Files\Altiris\InventoryRuleManagement\Web\web.config
  • \Program Files\Altiris\InventoryRuleManagement\Web\Agent\web.config

 

Isolate the Inventory Rule Management Web serviceIsolating the Inventory Rule Management Web service provides a mechanism of throttling the load caused by the inventory rule retrieval process.  The following steps provide the technique to accomplish this using IIS 6 which is installed with Windows 2003.

 

Set the master request queue limit

  1. Open IIS manager on the Notification Server.  Start > Program Files > Administrative Tools > Internet Information Services (IIS) Manager.
  2. Right-click Application Pools > Properties > Performance tab.
  3. Ensure that the Limit the kernel request queue option is enabled and set to 1,000 (this is the default value for a clean install of IIS 6).
  4. Click OK.


Define a new application pool

  1. Right-click Application Pools > New > Application Pool.
  2. In the new window, name the pool. Recommended name is "Inventory Rule Management".
  3. Keep the default pool settings and click OK.
  4. Right-click the newly created pool and choose Properties > Performance tab.
  5. Check the option Limit the kernel request queue and set it to 200.  **Note: If you see compression errors in the log files, up this setting to 500.
  6. Click OK.


Reassign the InventoryRuleManagement virtual directory to the new application pool

 

  1. Right-click Default Web Site > Altiris > InventoryRuleManagement.
  2. Choose the new application pool from the drop down interface (Application pool)
  3. Click OK.


     

Restart the IIS service

Go to Start > Run > IISReset.exe > OK.

Expected Results

The Notification Server console performance should increase, and server load should drastically decrease in comparison to previous PMImport release days. IIS logs should reflect a larger number of 503 error messages in reference to AgentRuleData.ashx requests. This is the intended behavior, and the agents will instead try back in a few minutes. The kernel request queue limit of 200 is appropriate for most environments. It may be adjusted to between 100 and 500.  Increasing the limit will slightly increase the speed at which vulnerability scanning data is distributed at the expense of additional CPU and memory consumption. 
 

Potential Issues with this technique

The repercussions of a executing a repair on the Patch Management .msi files has not been evaluated. The same concern exists when Patch Management Solution for Windows 6.2 is released and an upgrade from 6.1 to 6.2 is then performed. This article will be updated once an .msi repair evaluation has been completed.

Note: Customers should reverse the application pool assignment on the InventoryRuleManagement virtual directory, prior to performing a repair or upgrade of any Patch Management based solution. This concern will be negated upon final confirmation from development. (This step may not be necessary in some upgrade cases, after the upgrade has completed check the Application Pool to see if has changed to Default. If it has change it back to Inventory Rule Management.)

***Note for Patch Management 7.1+ users:   This has not been tested in 7.1+.  Results have been mixed in customer's environments and is not recommended by the Patch Development team. With the version of IIS used with 7.1+ the allocation of resources is handled better than in the 1.1.4322 .NET version.  It is suggested using /3GB /UserVA switch first to try to help memory (see KM Article ID: HOWTO4400), but there is no known reason that it would cause a problem. 

Symantec would suggest that customers start without setting this up and it could be added at a future time if needed.  If errors in IIS are seen place the InventoryRuleManagement segment back into the Default AppPool