ESXi host fails with PSOD "#PF Exception 14 in world xxxx:NetWorld-VM-IP addr PTE"

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

Symptoms:

• NSX version installed NSX-T Data Center 4.2.0.x - 4.2.1.x
• PSOD Screen trace shows these error strings
• "ESXi host fails with PSOD "#PF Exception 14 in world xxxx:NetWorld-VM-IP addr PTE" and "involved in panic: [nsxt-vsip-24302014 Version 1.0.0-0 RELEASEbuild-24302014]"

PSOD conditions and configuration observed:

• Actively using IDFW implemented firewall rules in the NSX Distributed Firewall
• IDFW enabled on clusters where ESXi PSOD occurred (GI or Log Scraping configured)
• Vmotion for some filters running simultaneously as IDFW rule lookup for other filters on the ESX host

Full Trace:
var/core/vmkernel-zdump.log 
####-##-##T17:15:31.649Z cpu104:2098242)Cleanup and detach filter, nic-########-eth0-vmware-sfw.2(355) at 0x433be35b6e20 (=73924496551456)
####-##-##T17:15:31.658Z cpu44:13664825)World: 3355: PRDA 0x42004b000000 ss 0x0 ds 0x750 es 0x750 fs 0x0 gs 0x0
####-##-##T17:15:31.658Z cpu44:13664825)World: 3357: TR 0x768 GDT 0xfffffffffca02888 (0xffff) IDT 0xfffffffffc408000 (0xffff)
####-##-##T17:15:31.658Z cpu44:13664825)World: 3359: CR0 0x80050033 CR3 0x584f98000 CR4 0x156668
####-##-##T17:15:31.658Z cpu104:2098242)Free kif empty_filter(355) at 0x433be35b6e20
####-##-##T17:15:31.692Z cpu44:13664825)Backtrace for current CPU #44, worldID=13664825, fp=0x433e32af1150
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c999db0:[0x420024360a27][e-mail]#1.0.8.0.24302014+0x1bb stack: 0x433e32aab8c8, 0x0, 0x453a0c999dc8, 0x433be349ee40, 0x50100000000
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c99a2b0:[0x420024358522][e-mail]#1.0.8.0.24302014+0x13ff stack: 0xac09d38232aab890, 0x453a0c99aed0, 0x1, 0x453a0c99ab00, 0x10c99a9a2
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c99a800:[0x4200242edf10][e-mail]#1.0.8.0.24302014+0x3bd stack: 0x0, 0x453a0c99a910, 0x453a0c99ae50, 0x0, 0x433d00000002
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c99ad10:[0x4200243005a9][e-mail]#1.0.8.0.24302014+0x34c6 stack: 0x14, 0x45ba2d3c52c8, 0x1, 0x0, 0x0
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c99afe0:[0x420024386603][e-mail]#1.0.8.0.24302014+0x7f8 stack: 0x0, 0x453a0c99b140, 0x0, 0x453a0c99b818, 0x0
####-##-##T17:15:31.692Z cpu44:13664825)0x453a0c99b300:[0x4200242978ce][e-mail]#1.0.8.0.24302014+0x67 stack: 0x2c, 0x2c, 0x433be335f568, 0x4200242742d8, 0x100000000
####-##-##T17:15:31.710Z cpu44:13664825)ESC[45mESC[33;1mVMware ESXi 8.0.3 [Releasebuild-24280767 x86_64]ESC[0m
#PF Exception 14 in world 13664825:NetWorld-VM- IP 0x420024360a27 addr 0x47b PTEs:0x0; <---------------Matched Log Snip

Module(s) involved in panic: [nsxt-vsip-24302014 Version 1.0.0-0 RELEASEbuild-24302014] <---------------Matched Log Snip
####-##-##T17:15:31.710Z cpu44:13664825)cr0=0x8001003d cr2=0x47b cr3=0x51864000 cr4=0x14616c
####-##-##T17:15:31.710Z cpu44:13664825)FMS=06/55/7 uCode=0x5003707
####-##-##T17:15:31.711Z cpu44:13664825)frame=0x453a0c999cf0 ip=0x420024360a27 err=0x0 rflags=0x10246
####-##-##T17:15:31.711Z cpu44:13664825)rax=0x0 rbx=0x453a0c99a398 rcx=0x4
####-##-##T17:15:31.711Z cpu44:13664825)rdx=0x0 rbp=0x433e32af1150 rsi=0x0
####-##-##T17:15:31.711Z cpu44:13664825)rdi=0x433e32a00030 r8=0x55415150 r9=0x433e32aa4c40
####-##-##T17:15:31.711Z cpu44:13664825)r10=0x433e32a00e58 r11=0x0 r12=0x0
####-##-##T17:15:31.711Z cpu44:13664825)r13=0x0 r14=0x453a0c99a33c r15=0x433be349ee40
####-##-##T17:15:31.711Z cpu44:13664825)pcpu:0 world:2099258 name:"tq:VSAN_0x4341e53251c0_Owner" (S)
####-##-##T17:15:31.711Z cpu44:13664825)pcpu:1 world:2098446 name:"vmnic3-pollWorld-33-0x4301558f5a80" (S)


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

Environment

VMware NSX 4.2.0 to 4.2.1 

Cause

Vmotions are happening on the system causing filter destroy along with simultaneous IDFW rule lookups for other filters, eventually causing this PSOD

Resolution

If you run into this PSOD, please create a support request case with Broadcom so that we can validate.

Fixed in this version of NSX:
Upgrade to NSX 4.2.2.

Workaround:
Workaround is to disable IDFW