Network timeouts or packet drops with VMware Tools 11.x with Guest Introspection Driver
search cancel

Network timeouts or packet drops with VMware Tools 11.x with Guest Introspection Driver

book

Article ID: 343520

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vDefend Firewall

Issue/Introduction

Symptoms:

  • Virtual Machine is using VMware Tools 11.x with Network Introspection driver (vnetWFP.sys) installed.
  • Users might experience packet loss (more than 1%) or other loss of network performance.
  • ICMP connectivity tests (e.g. ping) are reporting timeouts.
  • If debug logs are enabled for vnetWFP, (Please refer KB:339565 for enabling debug logs), you may see following entries in either debug viewer or vmware.log (depending on configuration)

DEBUG: ALEInspectInjectComplete : Packet injection status is : c000021b

Environment

VMware vSphere ESXi 7.0.0
VMware Tools 11.x
VMware vSphere ESXi 6.5
VMware NSX-T Data Center
VMware vSphere ESXi 6.7
VMware NSX-T Data Center 3.x

Cause

Packet drop is seen due to intermittent failure reported by the Microsoft WFP packet injection API. 

Resolution

Upgrade to VMTools 12+

Workaround:

Until the issue is fixed,  with VMTools 11.2.6, following registry setting can be used to disable the ICMP/UDP protocol support for the GI :

1) Click Start > Run, type regedit, and click OK. The Registry Editor window opens.

 

2) Create the following key using the registry editor:

HKEY_LOCAL_Machine\SYSTEM\CurrentControlSet\services\vnetwfp\parameters

 

3) Create the following DWORD value under the newly created parameters key:

Note: Ensure that Hexadecimal is selected when putting in these values. The ‘0x’ means hexadecimal, you should not enter ‘0x’ while adding the values in the registry. Selecting hexadecimal is enough. 

 

Name: ksam_otorp

Type: DWORD

 

Possible Values for ksam_otorp:

0x400 - To disable filtering of ICMP messages

0x800 - To disable filtering of UDP messages

0x0 - To disable filtering of both ICMP and UDP messages

0xC00 - To enable filtering of ICMP and UDP messages

 

4) User can delete the key to get the default behavior (i.e. all protocols enabled). On VMTools 11.2.6, vnetWFP needs to be reloaded after deleting the registry.
 

 

Customer Use CaseRecommended Registry SettingComments
Non NSX customers who have installed GI File and Network Introspection drivers.0x0This use case does not require GI introspection drivers and may even uninstall both the file and network introspection component.
NSX customers not using IDFW or NSX-Intelligence feature and also not using third party AV powered by Guest Introspection.0x0This use case does not require GI introspection drivers and may even uninstall both the file and network introspection component.
NSX customers not using IDFW or NSX-Intelligence feature. But are using third party AV.0x0This use case requires only the file introspection component and does not require GI network introspection driver and may safely uninstall the network introspection component.
NSX customers using NSX IDFW functionality in NSX-V or NSX-T environment.0x400Result of setting this registry, we would not be enforcing IDFW rules configured for ICMP protocol.
NSX customers using NSX Intelligence feature in NSX-T environment.0x400There would be no significant impact on NSX Intelligence feature while setting this registry settings.

 

 

Please note:

By default without the registry setting all the protocol messages are supported in GI platform, nothing is disabled.

Disabling the above messages in the GI platform will impact the GI based networking features like NSX IDFW, NSX Intelligence. Registry only disables these protocol for the GI platform, general ICMP/UDP messages continue to work

______________________________________

you can also run the following in command prompt

sc query vnetwfp

to see the status of Guest Introspection Driver and

sc stop vnetwfp

to stop the service and if the timeouts cease Guest Introspection is the root

if the registry workaround does not work and Guest Introspection is not necessary on this VM you can rename the vnetwfp service in system32 with _old at the end and then kill the service using

sc stop vnetwfp


Additional Information

The attached PR2470223-RCA.pdf contains preliminary investigation information about the cause of this issue.

Guest Introspection Network Driver (vnetWFP.sys) may fail to load (after reboot) when VMware Tools is upgraded while Windows defender is active


Impact/Risks:

NSX customers (especially those using IDFW and NSX-Intelligence features) and have done complete installation of VMTools 11.0.0+ on Windows endpoints may face this issue. 

NSX Agentless Antivirus uses File Introspection driver (vsepflt). NSX Identity Firewall, NSX Intelligence use Network Introspection driver (vnetWFP). This issue only concerns the NSX Network Introspection driver(vnetWFP).

The severity of the issue has been reduced (by a third in test environment) in VMTools 11.1.5 by introducing retry logic. (Note this is not a fix nor a workaround)

 

Attachments

PR2470223-RCA get_app