ESxi host running on version 6.5 or 6.7 crashes with a PSOD (purple screen of death) - PF Exception 14 in world <xxxx>:vmnic1-pollW IP

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi host running on 6.5 or 6.7 crashes with a PSOD (purple screen of death) - PF Exception 14 in world xxxxx:vmnic1-pollW IP
• The PSOD (purple screen of death) backtrace entries will be similar to

 2019-11-12T05:17:47.029Z cpu1:2097828)@BlueScreen: #PF Exception 14 in world 2097828:vmnic1-pollW IP     0x41800485dadc addr 0x20 PTEs:0x8073ba4023;0x8073ba5023;0x8073ba6023;0x0;
2019-11-12T05:17:47.029Z cpu1:2097828)Code start: 0x418004600000 VMK uptime: 6:16:11:10.061
2019-11-12T05:17:47.029Z cpu1:2097828)0x451acc11bd98:[0x41800485dadc]vmk_PktFrameLenSet@vmkernel#nover+0x0 stack: 0x41800482c77c
2019-11-12T05:17:47.030Z cpu1:2097828)0x451acc11bda0:[0x41800503c300]qfle3_rxeof@(qfle3)#<None>+0x1109 stack: 0x1
2019-11-12T05:17:47.030Z cpu1:2097828)0x451acc11bf00:[0x4180050421de]qfle3_netpoll_rxcb@(qfle3)#<None>+0x21b stack: 0x8000000000001054
2019-11-12T05:17:47.030Z cpu1:2097828)0x451acc11bf70:[0x41800485f804]NetPollWorldCallback@vmkernel#nover+0x199 stack: 0x14
2019-11-12T05:17:47.031Z cpu1:2097828)0x451acc11bfe0:[0x41800490eaa2]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0
2019-11-12T05:17:47.034Z cpu1:2097828)base fs=0x0 gs=0x418040400000 Kgs=0x0

• In the ESXi var/log/vmkernel.log you will see entries as below

 2019-11-12T05:17:46.961Z cpu6:2097292)qfle3: qfle3_rq_alloc:325: [vmnic1] Rxq 2 is leading RSS with 4 RSS queues.
[7m2019-11-12T05:17:46.972Z cpu1:2097828)WARNING: qfle3: ecore_queue_comp_cmd:4932: Bad MC reply 1 for queue 2 in state 1 pending 0x0, next_state 9
[7m2019-11-12T05:17:46.972Z cpu1:2097828)WARNING: qfle3: qfle3_sp_event:18842: [vmnic1] unexpected completion (1) on fp[2]
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: ecore_queue_comp_cmd:4932: Bad MC reply 10 for queue 2 in state 1 pending 0x2, next_state 2
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: qfle3_sp_event:18842: [vmnic1] unexpected completion (6) on fp[2]
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: ecore_queue_comp_cmd:4932: Bad MC reply 7 for queue 2 in state 1 pending 0x2, next_state 2
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: qfle3_sp_event:18842: [vmnic1] unexpected completion (2) on fp[2]
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: ecore_queue_comp_cmd:4932: Bad MC reply 9 for queue 2 in state 1 pending 0x2, next_state 2
[7m2019-11-12T05:17:46.973Z cpu1:2097828)WARNING: qfle3: qfle3_sp_event:18842: [vmnic1] unexpected completion (7) on fp[2]

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Cause

This occurs due to driver version mismatch  that are loaded other than qfle3.

Resolution

• Upgrade or Install the below version qfle3 drivers to fix the issue  

qfle3-1.0.86.0

     qfle3i_1.0.23.0
     qfle3f_1.0.68.0
     qcnic_1.0.26.0

• Reboot the host for the upgrade to be successful 

       
Download qfle drivers For ESXi 6.7 from Broadcom Downloads page. ESXi 6.7 Drivers & Tools

 

Go to Drivers & Tools tab.

Search for qfle driver.

Download qfle drivers For ESXi 6.5 from Broadcom Downloads page ESXi 6.5 Drivers & Tools

 

Go to Drivers & Tools tab.

Search for qfle driver.

Workaround:
To workaround this issue disable qfle3i,qfle3f and qcnic. Use the following command and reboot the ESXi host.
esxcfg-module -d qfle3i
esxcfg-module -d qfle3f
esxcfg-module -d qcnic

This will load only qfle3 driver.

Additional Information

Impact/Risks:
None