ESXi host fails with PSOD "#PF Exception 14 in world xxxx:vmnic0-pollW IP" during vMotions in an NSX-T Environment

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX version installed NSX-T Data Center 2.5.x, 3.0.x, 3.1.x and 3.2.0
• ESXi Host fails with PSOD with "PF Exception 14 in world xxxxxx:vmnic0-pollW IP"
• ESXi Host fails with PSOD with "PF Exception 14 in world xxxxxxxx:NetWorld-VM- IP
• This issue is observed when vMotions occur in the NSX-T environment with Firewall applied to vNICs, following are some of the probable scenarios:
  • Migration of multiple VMs with each VM comprising one or more vNICs
• Multiple IP sets configured dynamically per rule

 

The scenarios above may lead to PSOD with the following backtrace:
 

Panic Message: @BlueScreen: #PF Exception 14 in world 81525951:NetWorld-VM- IP 0x420013dc2b27 addr 0x10

Backtrace:

 0x453a7ea1ae88:[0x420013dc2b27]rn_match_int@(nsxt-vsip-18504670)#<None>+0x27 stack: 0x113d9ecb2, 0x2107ea19ea2, 0x33c0790a, 0x453a00000000, 0x0

 0x453a7ea1ae90:[0x420013d81528]pfr_update_stats@(nsxt-vsip-18504670)#<None>+0x9d stack: 0x2107ea19ea2, 0x33c0790a, 0x453a00000000, 0x0, 0x4320a7469b38

0x453a7ea1af00:[0x420013d195a7]pf_test@(nsxt-vsip-18504670)#<None>+0x1784 stack: 0x0, 0x0, 0x800000002, 0x45bda3fa1bfa, 0x45ba00000000

 

OR:
 

Panic Message: @BlueScreen: #PF Exception 14 in world 2111681:vmnic0-pollW IP 0x42001730baff addr 0x10

Backtrace:

 0x453b45e9ac08:[0x42001730baff]rn_match_int@(nsxt-vsip-18934158)#<None>+0x27 stack: 0x172e7c72, 0x21045e94c09, 0x1f8012ac, 0x453b00000000, 0x0

 0x453b45e9ac10:[0x4200172ca4e8]pfr_update_stats@(nsxt-vsip-18934158)#<None>+0x9d stack: 0x21045e94c09, 0x1f8012ac, 0x453b00000000, 0x0, 0x4389cac0b808

0x453b45e9ac80:[0x4200172625e1]pf_test@(nsxt-vsip-18934158)#<None>+0x17be stack: 0x0, 0x0, 0x800000001, 0x45e1102a567a, 0x45e000000000

0x453b45e9ae80:[0x4200172e9f8f]PFFilterPacket@(nsxt-vsip-18934158)#<None>+0x754 stack: 0x0, 0x453b45e9afc0, 0x0, 0x453b45e9afc0, 0x438900000000

 

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 2.5.x

Cause

A PSOD can occur when the following two changes are happening concurrently to the vMotioned VM(s).
1. Reconfiguration of an address set either an addition or removal of existing members in the dynamically configured rule
2. Packet processing of a packet that hits a rule that contains the address set being updated in step 1

Resolution

This is fixed in VCF release 4.4.1 BOM containing NSX-T 3.1.3.7.4, 3.2.1 and later releases 

Workaround:

To work around this, you can limit the number of vMotions by setting DRS in the cluster to manual. 
Refer to this link to on how configure DRS as Manual