Third Party Service VM reports error "dvfilter_init failed" after host reboot

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• A third party service VM may not function after its native ESXi host was rebooted.
• You may see service errors similar to: Failed initializing communication with ESX Host (dvfilter_init failed).
• When running summarize-dvfilter command, you see that the "slowPathID" is not defined:

[root@:~] summarize-dvfilter
Fastpaths:
agent: dvfilter-faulter, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter
agent: ESXi-Firewall, refCount: 3, rev: 0x1010000, apiRev: 0x1010000, module: esxfw
agent: dvfilter-generic-vmware, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter-generic-fastpath
agent: dvfilter-generic-vmware-swsec, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfilter-switch-security
agent: bridgelearningfilter, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: vdrb
agent: dvfg-igmp, refCount: 1, rev: 0x1010000, apiRev: 0x1010000, module: dvfg-igmp
agent: vmware-sfw, refCount: 4, rev: 0x1010000, apiRev: 0x1010000, module: vsip
agent: serviceinstance-4, refCount: 6, rev: 0x1010000, apiRev: 0x1010000, module: vsip

Slowpaths:

Filters:
world 0
port 33554438 vmk0
vNic slot 0
name: nic-0-eth4294967295-ESXi-Firewall.0
agentName: ESXi-Firewall
state: IOChain Attached
vmState: Detached
failurePolicy: failOpen
slowPathID: none
filter source: Invalid
port 33554439 vmk1
vNic slot 0
name: nic-0-eth4294967295-ESXi-Firewall.0
agentName: ESXi-Firewall
state: IOChain Attached
vmState: Detached
failurePolicy: failOpen
slowPathID: none
filter source: Invalid

 

Cause

The issue occurs because the SVMs permissions (provided by EAM during deployment) become absent after the host reboot, as they do not reside natively on the host.

Resolution

This issue is resolved in VMware ESXi 6.5 update 1 and ESXi 6.0 P6., available at VMware Downloads.

To work around this issue, if you do not want to upgrade:

• Verify the permissions of the VM when connection issue is observed:

From an ESX shell: 
Run ps -Z | grep 'vmx' and locate the line that corresponds to the VM in question. Note the value of the last column: security domain ID.
Run secpolicytools -d and look for permissions related to dvfilter and vsocket.

• If you see an issue with permissions, restart the SVM.
• If this issue is only reproducible when the VM does not have the dvfilter permissions, please try the following:

Power off service VM.
Run the attached python script with parameters "-v " and "-p "
Power on the VM.