File Introspection Driver (vsepflt) loses communication with host component (nsx-context-mux) after Windows Update

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

File Introspection Driver vsepflt loses communication to the host component Mux. This impacts all guest related file/file-less, login/logoff events delivered to the host components. Consumers of the guest events including Intelligence, MPS and IDFW feature are affected by this issue. 

Connection breakage during VMCI upgrade  through Windows Update. This can be verified with below log message found in vmware.log for the Guest VM :

 

xxxx-xx-xxTxx:xx:xx.xxx In(05) vcpu-0 - Guest: vsep: AUDIT: VFileSocketMgrCloseSocket : Mux is disconnected
xxxx-xx-xxTxx:xx:xx.xxx In(05) vcpu-2 - Guest: vsep: AUDIT: vsepAuditSvmConnectivity : Lost connectivity to Solution(6341068275337662464), muxStatus: 0xc000020c
xxxx-xx-xxTxx:xx:xx.xxx In(05) vcpu-0 - Guest: Driver=vmci, Version=9.8.28.0 - unloaded

 

Users may see File introspection Count as 0

Environment

• VM Tools version 12.0 or later with Windows Update.
• Specific to Windows VM only.
• Happens only VMCI driver upgrade happens through Windows Update.

SSP>= 5.0

Cause

File Introspection Driver vsepflt has dependency on VMCI driver which is also installed through VMware tools.  Starting from Tools Version 12.0, VMCI driver have the options of getting updated through the Windows Update while vsepflt requires installation through tools installation.  If the VMCI driver gets upgraded through Windows Update, vsepflt loses communication to the host due to the driver restart. vsepflt fails to reestablish communication through the newly installed VMCI driver.

Resolution

Workaround 1: Restarting the VM after Windows Update fixes the issue as the vsepflt can establish fresh communication at boot time.

Workaround 2: This method can be done without a VM reboot by restarting the vsock subsystem. Recommended to perform it during a maintenance window.
- Open command line or Powershell in Windows with administrative privileges.
- Stop and start the service.
sc stop vsock
sc start vsock
- If for any reason sc commands don't return any output it means probably the domain settings disallow it. For that follow these steps
- Navigate to Device Manager, System Devices
- Under there, disable/renable VMCI Bus Device and VMCI Host Device highligted below:

- You can verify that the VMCI had been reinitialized by checking Event Viewer --> System Events. You will see VMCI initialization messages.  

NOTE: This issue  fixed in VMTools  Version 13.1.0