Windows BSOD with vnetflt.sys from old VMware Tools package

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• A windows machine experiences a BSOD.
• The Windows BSOD screen displays a message similar to:
  
  If you call a support person, give them this info: 
  Stop code: PAGE FAULT IN NONPAGED AREA
  What failed: vsepflt.sys
   
• The windbg crash call stack contains entries similar to:
  
  STACK_TEXT:  
  ffffd000`32cbc108 fffff803`f2fe7910 : 00000000`00000050 ffffe000`d4f6e000 00000000`00000000 ffffd000`32cbc2a0 : nt!KeBugCheckEx
  ffffd000`32cbc110 fffff803`f2e4bfb9 : 00000000`00000000 ffffe000`d4f6e000 ffffd000`32cbc2a0 ffffe000`d4f6e000 : nt!MiSystemFault+0x1048
  ffffd000`32cbc1a0 fffff803`f2f4eb9d : 00000000`00000000 ffffe000`d24fe100 ffffe000`d4f6df00 fffff801`c0168cb3 : nt!MmAccessFault+0x219
  ffffd000`32cbc2a0 fffff801`c01635b5 : 00000000`6c694656 fffff801`c0181e40 00000000`00000100 00000000`00000000 : nt!KiPageFault+0x31d
  ffffd000`32cbc430 fffff801`c0162665 : ffffe000`d12f61b0 fffff801`c069a910 00000000`00000001 00000000`00000000 : vsepflt+0xf5b5
  ffffd000`32cbc460 fffff801`c06962c5 : ffffe000`cecb51b0 fffff801`c069a970 fffff801`c069a910 00000000`00000000 : vsepflt+0xe665
  ffffd000`32cbc490 fffff801`c0697a8d : ffffe000`d0ee0df0 ffffe000`d0de5f50 ffffe000`cecb5060 ffffe000`d0ee0df0 : vnetflt+0x12c5
  ffffd000`32cbc4d0 fffff801`c0698958 : ffffe000`d0ee0df0 ffffe000`d1408230 ffffe000`d1b3c580 ffffe000`d53bacf0 : vnetflt+0x2a8d
  ffffd000`32cbc500 fffff801`c0696d0d : ffffe000`d0ee0df0 ffffe000`d0ee0df0 ffffe000`d0de5f50 00000000`00000000 : vnetflt+0x3958
  ffffd000`32cbc530 fffff801`c00896a5 : ffffe000`d0de5ec0 ffffe000`cf738220 00000000`00000010 00000000`00000002 : vnetflt+0x1d0d

Cause

This issue occurs due to an outdated vnetflt.sys driver that was included from a previously outdated VMware Tools package.

Consider this use case:
You installed the VMware Tools NetworkIntrospection feature. After explicitly removing this feature and deciding to keep only the FileIntrospection feature when upgrading VMware Tools later.

Due to a long-time bug in the tools installer, after the NetworkIntrospection feature was removed in the MSI database, the feature's vnetflt or vnetWFP (for tools version >= 10.2.5 on Windows 7 & above) driver service still exists. This old vNet driver may cause the Windows BSOD when co-existing with the FileIntrospection feature's current vsepflt driver.

Note: This issue is seen on Windows 10, Windows Server 2012 R2 with VMware Tools 11.0.x, but can happen in other cases.

Resolution

To resolve this issue, remove the old vNet driver service from either the Guest OS or the Host side.

From the Guest OS

1. Download the attached RemoveOldVNetDriver.ps1 file in this KB.
2. Read the description inside the attached RemoveOldVNetDriver.ps1 file.
3. Run the script in a Windows PowerShell (Admin) session.
4. Check its output.

From the Host

1. Upgrade to your desired VMware Tools version in the way you normally do with FileIntrospection and NetworkIntrospection features included.
   Note: Do not remove the two features and do not suppress reboot.
   
   For example: The installer parameter for upgrading to VMware Tools version 11.0.x: /s /v"-qn ADDLOCAL=All REMOVE=BootCamp,Hgfs,AppDefense".
    
2. After performing the VMware Tools upgrade, remove the NetworkIntrospection feature and its current vNet driver service.
   
   Installer parameter to use: /s /v"-qn REMOVE=NetworkIntrospection"
   
   Note: In future VMware Tools upgrade, you can simply use installer parameter: /s /v"-qn".

Another method is to navigate to the Programs and Features > VMware Tools > right click Change > Next > select Modify > Next > under VMCI Driver > select NSX Network Introspection Driver > select X Entire feature will be unavailable > Next > Change > Finish