Third party applications, Windows kernel memory and Spectrum
Article ID: 32146
Updated On:
Products
Issue/Introduction
Windows operating systems use two finite memory pools for kernel components, based on the amount of physical memory on the machine and the architecture of the Windows version. These are the paged pool, which can be paged out to disk if physical memory is low, and the much smaller non-paged pool (NPP), which guarantees that memory allocated will remain in physical memory. Once non-paged memory has been allocated it cannot be freed. For more information on the Windows kernel address space, see the references below.
What does this have to do with Spectrum? The SpectroSERVER and Archive Manager can be indirectly impacted when non-paged pool memory is exhausted. Third party applications (often antivirus, backup and Host Intrusion Prevention (or HIPS) applications) can consume all available memory in the non-paged pool (NPP). When antivirus and host intrusion applications scan the system, they cache data in the NPP. When most of the non-paged pool memory has been allocated, the Windows kernel (srv.exe process) can't allocate more for itself or for application handles and drivers. This can result in SpectroSERVER and Archive Manager crashes, as well as performance problems in the TCPIP stack.
Some examples (not an exhaustive list) of applications that could consume kernel memory:
Veritas
ARCserve Backup Agent for Open Files or ARCserve Open File Agent
Inoculan
McAfee antivirus and/or Host Intrusion Prevention System
Trend Micro
Symantec antivirus
SYMPTOMS:
Remote desktops stop responding. The system is slow overall.
The Windows Application event log might include these events:
Event ID: 2022
Source: Srv
Type: Error
Description:
Server was unable to find a free connection <number> times in the last <number> seconds.
Event ID: 2021
Source: Srv
Type: Error
Description:
Server was unable to create a work item <number>> times in the last <number> seconds.
Event ID: 2019
Source: Srv
Type: Error
Description:
The server was unable to allocate from the system nonpaged pool because the pool was empty.
It is also possible for third party applications to exhaust the paged pool, in which case you will see this event:
Event ID: 2020
Source: Srv
Type: Error
Description:
The server was unable to allocate from the system paged pool because the pool was empty.
The Spectrum $SPECROOT/SS/VNM.OUT file and Spectrum Control Panel could show this message: "Fatal error unable to allocate heap"
The problem is more likely to occur on pre-Vista and 32-bit pre Windows Server 2008 versions of Windows. The problem also might start happening after adding the /3GB switch to the boot.ini. This cuts the available kernel memory in half (see the technet blog).
Although there are known non-paged pool leaks in some antivirus and Host Intrusion Prevention software releases, it is the overall allocation of non-paged pool memory that causes the problem. In other words, a leak could be responsible, but your third party software might be working as designed and still exhaust all available NPP memory.
Environment
Component: SPCAPP
Resolution
Follow Microsoft KB 177415 to use poolmon to determine which driver is using most of the pool and/or leaking it:
How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
Try disabling the relevant service(s). Sometimes it takes more than disabling services. Microsoft KB 816071 explains how to disable kernel mode filters.
Make sure your third-party software is up to date. Contact your system administrator to update the relevant driver(s).
Contact the software manufacturer's support.
Consider upgrading to 64-bit Windows Server 2008.
Additional Information
REFERENCES:
Memory Management - Demystifying /3GB
http://blogs.technet.com/b/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx
Memory Management - Understanding Pool Resources
http://blogs.technet.com/b/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
An Overview of Troubleshooting Memory Issues
http://blogs.technet.com/b/askperf/archive/2008/01/25/an-overview-of-troubleshooting-memory-issues.aspx
Troubleshooting Server Hangs
http://blogs.technet.com/b/askperf/archive/2007/09/25/troubleshooting-server-hangs-part-one.aspx
Memory Management - x86 Virtual Address Space
http://blogs.technet.com/b/askperf/archive/2007/09/28/memory-management-x86-virtual-address-space.aspx
Event ID 2019 or 2020 or "Insufficient System Resources" error returned when logging on
Microsoft KB 272568
High non-paged pool memory usage on VirusScan Enterprise 8.7i
McAfee KB66905
Not enough server storage is available to process this command (when clients attempt to access network shares) note:
McAfee KB 59932
Your Windows Server 2003-based or Windows 2000-based computer occasionally becomes unresponsive, and you receive an Event ID 2019 error message in the System log
Microsoft KB 888928
Feedback
