A memory leak no longer occurs after uninstalling Endpoint Protection
Article ID: 165245
Updated On:
Products
Issue/Introduction
You experience a memory leak on a system with Symantec Endpoint Protection (SEP), i.e. the amount of available RAM keeps decreasing until the point of failure. The issue disappears after uninstalling SEP.
After generating a complete memory dump of the system, opening it using the Windows Debugger (WinDBG) and issuing the !vm command to display the virtual memory usage overview, high non-paged pool memory usage is shown:
1: kd> !vm 1
Page File: \??\C:\pagefile.sys
Current: 6554624 Kb Free Space: 6534292 Kb
Minimum: 6554624 Kb Maximum: 6554624 Kb
Physical Memory: 1572750 ( 6291000 Kb)
Available Pages: 470718 ( 1882872 Kb)
ResAvail Pages: 728528 ( 2914112 Kb)
[...]
NonPagedPool Usage: 797745 ( 3190980 Kb)
NonPagedPool Max: 1168888 ( 4675552 Kb)
PagedPool 0 Usage: 70714 ( 282856 Kb)
PagedPool 1 Usage: 8606 ( 34424 Kb)
PagedPool 2 Usage: 3953 ( 15812 Kb)
PagedPool 3 Usage: 3959 ( 15836 Kb)
PagedPool 4 Usage: 3819 ( 15276 Kb)
PagedPool Usage: 91051 ( 364204 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
[...]
On further investigation, most of this is used by fwpkclnt.sys (Microsoft's FWP/IPsec Kernel-Mode API driver):
1: kd> !poolused 3 -t 5
..
Sorting by NonPaged Pool Consumed
NonPaged Paged
Tag Allocs Frees Diff Used Allocs Frees Diff Used
Fwpx 14869079 0 14869079 2854863168 0 0 0 0 WFP NBL tagged context , Binary: fwpkclnt.sys
WfpH 87858350 87792639 65711 9142496 0 0 0 0 WFP hash , Binary: netio.sys
NDCM 195461088 195460363 725 7427760 0 0 0 0 UNKNOWN pooltag 'NDCM'
File 30866724 30850476 16248 5412896 0 0 0 0 File objects
VM3D 42 6 36 3829664 3 0 3 160 Volume Manager , Binary: volmgr.sys
TOTAL 733844266 718740053 15104213 2924208544 265926928 265601188 325740 364486672
Displaying the verbose loaded modules information for fwpkclnt.sys indicates a more recent version may be available:
1: kd> lmvm fwpkclnt
Browse full module list
start end module name
fffff88001a15000 fffff88001a5f000 fwpkclnt (deferred)
Image path: \SystemRoot\System32\drivers\fwpkclnt.sys
Image name: fwpkclnt.sys
Browse all global symbols functions data
Timestamp: Sat Nov 20 01:21:37 2010 (4CE79321)
CheckSum: 0004AB00
ImageSize: 0004A000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
ImageSize: 0004A000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Environment
SEP 12.1, 14.0 or higher
Windows Server 2008 R2
Cause
A Bing search for fwpkclnt.sys 7601 site:support.microsoft.com language:en yields, among things, the following:
- KB2789378 ("Memory leak when an application uses the FwpsNetBufferListAssociateContext0 function in Windows 7 or Windows Server 2008 R2")
- KB2885980 ("FIX: A memory leak condition occurs when the FwpsAllocateCloneNetBufferlist() API is called")
- KB2918550 ("Computer leaks nonpaged pool memory when IPSEC traffic is configured to use AuthIP without encryption in Windows")
Respectively, these contain a November 29, 2012, October 9, 2013 and February 5, 2014 version of fwpkclnt.sys (Microsoft's FWP/IPsec Kernel-Mode API driver). As explained in the first two hotfixes' description, the issue occurs because fwpkclnt.sys does not free the context of a Windows Filtering Platform (WFP) callout driver or Network Driver Interface Specification (NDIS) Lightweight Filter (LWF) driver –abundantly used in any vendor's endpoint security software.
Resolution
Install Microsoft hotfix https://support.microsoft.com/en-us/help/2918550/computer-leaks-nonpaged-pool-memory-when-ipsec-traffic-is-configured-to-use-authip-without-encryption-in-windows, which includes the latest, Limited Distribution Release (LDR) version of fwpkclnt.sys that has been empirically proven to include the fix that resolves this issue.
Feedback
