A memory leak no longer occurs after uninstalling Endpoint Protection
search cancel

A memory leak no longer occurs after uninstalling Endpoint Protection

book

Article ID: 165245

calendar_today

Updated On:

Products

Endpoint Protection

Issue/Introduction

You experience a memory leak on a system with Symantec Endpoint Protection (SEP), i.e. the amount of available RAM keeps decreasing until the point of failure. The issue disappears after uninstalling SEP.

After generating a complete memory dump of the system, opening it using the Windows Debugger (WinDBG) and issuing the !vm command to display the virtual memory usage overview, high non-paged pool memory usage is shown:

1: kd> !vm 1
Page File: \??\C:\pagefile.sys
  Current:   6554624 Kb  Free Space:   6534292 Kb
  Minimum:   6554624 Kb  Maximum:      6554624 Kb
Physical Memory:          1572750 (    6291000 Kb)
Available Pages:           470718 (    1882872 Kb)
ResAvail Pages:            728528 (    2914112 Kb)
[...]
NonPagedPool Usage:        797745 (    3190980 Kb)
NonPagedPool Max:         1168888 (    4675552 Kb)
PagedPool 0 Usage:          70714 (     282856 Kb)
PagedPool 1 Usage:           8606 (      34424 Kb)
PagedPool 2 Usage:           3953 (      15812 Kb)
PagedPool 3 Usage:           3959 (      15836 Kb)
PagedPool 4 Usage:           3819 (      15276 Kb)
PagedPool Usage:            91051 (     364204 Kb)
PagedPool Maximum:       33554432 (  134217728 Kb)
[...]

On further investigation, most of this is used by fwpkclnt.sys (Microsoft's FWP/IPsec Kernel-Mode API driver):

1: kd> !poolused 3 -t 5
..
 Sorting by NonPaged Pool Consumed
 
                            NonPaged                                         Paged
 Tag       Allocs       Frees      Diff         Used       Allocs       Frees      Diff         Used
 Fwpx    14869079           0  14869079   2854863168            0           0         0            0    WFP NBL tagged context , Binary: fwpkclnt.sys
 WfpH    87858350    87792639     65711      9142496            0           0         0            0    WFP hash , Binary: netio.sys
 NDCM   195461088   195460363       725      7427760            0           0         0            0    UNKNOWN pooltag 'NDCM'
 File    30866724    30850476     16248      5412896            0           0         0            0    File objects 
 VM3D          42           6        36      3829664            3           0         3          160    Volume Manager , Binary: volmgr.sys
TOTAL   733844266   718740053  15104213   2924208544    265926928   265601188    325740    364486672

Displaying the verbose loaded modules information for fwpkclnt.sys indicates a more recent version may be available:

1: kd> lmvm fwpkclnt
Browse full module list
start             end                 module name
fffff88001a15000 fffff88001a5f000   fwpkclnt   (deferred)
    Image path: \SystemRoot\System32\drivers\fwpkclnt.sys
    Image name: fwpkclnt.sys
    Browse all global symbols  functions  data
    Timestamp:        Sat Nov 20 01:21:37 2010 (4CE79321)
    CheckSum:         0004AB00
    ImageSize:        0004A000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
    ImageSize:        0004A000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Environment

SEP 12.1, 14.0 or higher

Windows Server 2008 R2

Cause

A Bing search for fwpkclnt.sys 7601 site:support.microsoft.com language:en yields, among things, the following:

  • KB2789378 ("Memory leak when an application uses the FwpsNetBufferListAssociateContext0 function in Windows 7 or Windows Server 2008 R2")
  • KB2885980 ("FIX: A memory leak condition occurs when the FwpsAllocateCloneNetBufferlist() API is called")
  • KB2918550 ("Computer leaks nonpaged pool memory when IPSEC traffic is configured to use AuthIP without encryption in Windows")

Respectively, these contain a November 29, 2012, October 9, 2013 and February 5, 2014 version of fwpkclnt.sys (Microsoft's FWP/IPsec Kernel-Mode API driver). As explained in the first two hotfixes' description, the issue occurs because fwpkclnt.sys does not free the context of a Windows Filtering Platform (WFP) callout driver or Network Driver Interface Specification (NDIS) Lightweight Filter (LWF) driver –abundantly used in any vendor's endpoint security software.     

Resolution

Install Microsoft hotfix https://support.microsoft.com/en-us/help/2918550/computer-leaks-nonpaged-pool-memory-when-ipsec-traffic-is-configured-to-use-authip-without-encryption-in-windows, which includes the latest, Limited Distribution Release (LDR) version of fwpkclnt.sys that has been empirically proven to include the fix that resolves this issue.