Network timeouts or packet drops with VMware Tools 11.x with Guest Introspection Driver
Article ID: 343520
Updated On:
Products
Issue/Introduction
Symptoms:
- Virtual Machine is using VMware Tools 11.x with Network Introspection driver (vnetWFP.sys) installed.
- Users might experience packet loss (more than 1%) or other loss of network performance.
- ICMP connectivity tests (e.g. ping) are reporting timeouts.
- If debug logs are enabled for vnetWFP, (Please refer KB:339565 for enabling debug logs), you may see following entries in either debug viewer or vmware.log (depending on configuration)
DEBUG: ALEInspectInjectComplete : Packet injection status is : c000021b
Environment
VMware Tools 11.x
VMware vSphere ESXi 6.5
VMware NSX-T Data Center
VMware vSphere ESXi 6.7
VMware NSX-T Data Center 3.x
Cause
Packet drop is seen due to intermittent failure reported by the Microsoft WFP packet injection API.
Resolution
Workaround:
Until the issue is fixed, with VMTools 11.2.6, following registry setting can be used to disable the ICMP/UDP protocol support for the GI :
1) Click Start > Run, type regedit, and click OK. The Registry Editor window opens.
2) Create the following key using the registry editor:
|
HKEY_LOCAL_Machine\SYSTEM\CurrentControlSet\services\vnetwfp\parameters |
3) Create the following DWORD value under the newly created parameters key:
Note: Ensure that Hexadecimal is selected when putting in these values. The ‘0x’ means hexadecimal, you should not enter ‘0x’ while adding the values in the registry. Selecting hexadecimal is enough.
|
Name: ksam_otorp Type: DWORD |
Possible Values for ksam_otorp:
|
0x400 - To disable filtering of ICMP messages 0x800 - To disable filtering of UDP messages 0x0 - To disable filtering of both ICMP and UDP messages 0xC00 - To enable filtering of ICMP and UDP messages |
4) User can delete the key to get the default behavior (i.e. all protocols enabled). On VMTools 11.2.6, vnetWFP needs to be reloaded after deleting the registry.
| Customer Use Case | Recommended Registry Setting | Comments |
| Non NSX customers who have installed GI File and Network Introspection drivers. | 0x0 | This use case does not require GI introspection drivers and may even uninstall both the file and network introspection component. |
| NSX customers not using IDFW or NSX-Intelligence feature and also not using third party AV powered by Guest Introspection. | 0x0 | This use case does not require GI introspection drivers and may even uninstall both the file and network introspection component. |
| NSX customers not using IDFW or NSX-Intelligence feature. But are using third party AV. | 0x0 | This use case requires only the file introspection component and does not require GI network introspection driver and may safely uninstall the network introspection component. |
| NSX customers using NSX IDFW functionality in NSX-V or NSX-T environment. | 0x400 | Result of setting this registry, we would not be enforcing IDFW rules configured for ICMP protocol. |
| NSX customers using NSX Intelligence feature in NSX-T environment. | 0x400 | There would be no significant impact on NSX Intelligence feature while setting this registry settings. |
Please note:
By default without the registry setting all the protocol messages are supported in GI platform, nothing is disabled.
Disabling the above messages in the GI platform will impact the GI based networking features like NSX IDFW, NSX Intelligence. Registry only disables these protocol for the GI platform, general ICMP/UDP messages continue to work
______________________________________
you can also run the following in command prompt
sc query vnetwfp
to see the status of Guest Introspection Driver and
sc stop vnetwfp
to stop the service and if the timeouts cease Guest Introspection is the root
if the registry workaround does not work and Guest Introspection is not necessary on this VM you can rename the vnetwfp service in system32 with _old at the end and then kill the service using
sc stop vnetwfp
Additional Information
The attached PR2470223-RCA.pdf contains preliminary investigation information about the cause of this issue.
Guest Introspection Network Driver (vnetWFP.sys) may fail to load (after reboot) when VMware Tools is upgraded while Windows defender is active
Impact/Risks:
NSX customers (especially those using IDFW and NSX-Intelligence features) and have done complete installation of VMTools 11.0.0+ on Windows endpoints may face this issue.
NSX Agentless Antivirus uses File Introspection driver (vsepflt). NSX Identity Firewall, NSX Intelligence use Network Introspection driver (vnetWFP). This issue only concerns the NSX Network Introspection driver(vnetWFP).
The severity of the issue has been reduced (by a third in test environment) in VMTools 11.1.5 by introducing retry logic. (Note this is not a fix nor a workaround)
Attachments
Feedback
