ESXi host fails with PSOD "#PF Exception 14 in world xxxx:vmnic0-pollW IP" during vMotions in an NSX-T Environment

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• NSX version installed NSX-T Data Center 2.5.x, 3.0.x, 3.1.x and 3.2.0
• ESXi Host fails with PSOD with "PF Exception 14 in world xxxxxx:vmnic0-pollW IP"
• ESXi Host fails with PSOD with "PF Exception 14 in world xxxxxxxx:NetWorld-VM- IP
• This issue is observed when vMotions occur in the NSX-T environment with Firewall applied to vNICs, following are some of the probable scenarios:
  • Migration of multiple VMs with each VM comprising one or more vNICs
• Multiple IP sets configured dynamically per rule

 

The above scenarios above may lead to PSOD with the following backtrace:

Panic Message: @BlueScreen: #PF Exception 14 in world 81525951:NetWorld-VM- IP 0x420013dc2b27 addr 0x10

Backtrace:

 0x453a7ea1ae88:[0x420013dc2b27]rn_match_int@(nsxt-vsip-18504670)#<None>+0x27 stack: 0x113d9ecb2, 0x2107ea19ea2, 0x33c0790a, 0x453a00000000, 0x0

 0x453a7ea1ae90:[0x420013d81528]pfr_update_stats@(nsxt-vsip-18504670)#<None>+0x9d stack: 0x2107ea19ea2, 0x33c0790a, 0x453a00000000, 0x0, 0x4320a7469b38

0x453a7ea1af00:[0x420013d195a7]pf_test@(nsxt-vsip-18504670)#<None>+0x1784 stack: 0x0, 0x0, 0x800000002, 0x45bda3fa1bfa, 0x45ba00000000

OR:

Panic Message: @BlueScreen: #PF Exception 14 in world 2111681:vmnic0-pollW IP 0x42001730baff addr 0x10

Backtrace:

 0x453b45e9ac08:[0x42001730baff]rn_match_int@(nsxt-vsip-18934158)#<None>+0x27 stack: 0x172e7c72, 0x21045e94c09, 0x1f8012ac, 0x453b00000000, 0x0

 0x453b45e9ac10:[0x4200172ca4e8]pfr_update_stats@(nsxt-vsip-18934158)#<None>+0x9d stack: 0x21045e94c09, 0x1f8012ac, 0x453b00000000, 0x0, 0x4389cac0b808

0x453b45e9ac80:[0x4200172625e1]pf_test@(nsxt-vsip-18934158)#<None>+0x17be stack: 0x0, 0x0, 0x800000001, 0x45e1102a567a, 0x45e000000000

0x453b45e9ae80:[0x4200172e9f8f]PFFilterPacket@(nsxt-vsip-18934158)#<None>+0x754 stack: 0x0, 0x453b45e9afc0, 0x0, 0x453b45e9afc0, 0x438900000000

Environment

VMware NSX-T Data Center
VMware vSphere ESXi

Cause

A PSOD can occur when the following two changes are happening concurrently to the vMotioned VM(s):

1. Reconfiguration of an address set either an addition or removal of existing members in the dynamically configured rule
2. Packet processing of a packet that hits a rule that contains the address set being updated in step 1

Resolution

This is fixed in VCF release 4.4.1 BOM containing NSX-T 3.1.3.7.4 or 3.2.1 and later releases 

Workaround:

• To work around this, please limit the number of vMotions by setting the DRS in the cluster to manual. 
• Please refer to the attached link for more information on configuring DRS automation level to manual: Create a vSphere DRS Cluster