ESXi 5.x host with a Cisco fnic drivers experiences a purple diagnostic screen with Exception 14 and fnic driver errors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When your ESXi host uses a Cisco fnic driver, you experience a purple diagnostic screen with different errors:

• Immediately before the purple diagnostic screen, you see the messages:
  
  cpu8:2173)<7>fnic : 4 :: Device reset timed out
cpu8:2173)<7>fnic : 4 :: Returning from device reset FAILED
cpu8:2173)<7>fnic : 5 :: Device reset called FCID 0x20100, LUN 0x4e
cpu8:2173)<7>fnic : 5 :: TAG 0
cpu0:4424)<7>fnic : 5 :: dev reset cmpl recd. id 0 status FCPIO_SUCCESS
cpu8:2173)<7>fnic : 5 :: Returning from device reset SUCCESS
  
  
• The purple diagnostic screen displays messages similar to:
  
  
  • YYYY-MM-DDT15:03:03.078Z cpu2:4811)@BlueScreen: #PF Exception 14 in world 4811:helper31-13 IP 0x41803c2706bb addr 0x148
    YYYY-MM-DDT15:03:03.079Z cpu2:4811)Code start: 0x41803bc00000 VMK uptime: 9:00:00:16.695
    YYYY-MM-DDT15:03:03.080Z cpu2:4811)0x41224b2c7e60:[0x41803c2706bb]fnic_terminate_rport_io@<None>#<None>+0x9e stack: 0x1f
    YYYY-MM-DDT15:03:03.081Z cpu2:4811)0x41224b2c7eb0:[0x41803c0dfb14][email protected]#9.2+0x13f stack: 0x410001
    YYYY-MM-DDT15:03:03.083Z cpu2:4811)0x41224b2c7f60:[0x41803c0ea697][email protected]#9.2+0x11a stack: 0x0
    YYYY-MM-DDT15:03:03.084Z cpu2:4811)0x41224b2c7ff0:[0x41803bc3e129]helpFunc@vmkernel#nover+0x568 stack: 0x0
    YYYY-MM-DDT15:03:03.085Z cpu2:4811)0x41224b2c7ff8:[0x0]<unknown> stack: 0x0
    
    
  • cpu13:16580)pcpu 25 TLB NMI: 0x41225de9b6d0:[0x41801fd8989d][e-mail]#9.2+0x4 stac
    cpu13:16580)pcpu 25 TLB NMI: 0x41225de9b720:[0x418020131837][e-mail]#9.2.0.0+0x30
    cpu13:16580)pcpu 25 TLB NMI: 0x41225de9b770:[0x41802012d164][e-mail]#9.2.0.0+0x
    cpu13:16580)pcpu 25 TLB NMI: 0x41225de9b790:[0x41802012981d][e-mail]#9.2.0.0+0x18 s
    cpu13:16580)pcpu 25 TLB NMI: 0x41225de9b7d0:[0x41801fd6bd3c][email protected]#9.2+0x2b stac
    
    
  • YYYY-MM-DDT15:12:27.890Z cpu28:4124)@BlueScreen: #PF Exception 14 in world 4124:idle28 IP 0x41801c071028 addr 0xc70
    YYYY-MM-DDT15:12:27.890Z cpu28:4124)Code start: 0x41801ba00000 VMK uptime: 12:20:51:48.498
    YYYY-MM-DDT15:12:27.892Z cpu28:4124)0x4122407079f8:[0x41801c071028]fnic_host_end_tag@<None>#<None>+0x23 stack: 0x7e3e80e6a5a68
    YYYY-MM-DDT15:12:27.893Z cpu28:4124)0x412240707a78:[0x41801c06dcf3]fnic_fcpio_cmpl_handler@<None>#<None>+0xbca stack: 0x6a0a1db
    YYYY-MM-DDT15:12:27.894Z cpu28:4124)0x412240707ac8:[0x41801c06cad4]fnic_wq_copy_cmpl_handler@<None>#<None>+0xaf stack: 0x0
    YYYY-MM-DDT15:12:27.895Z cpu28:4124)0x412240707ae8:[0x41801c06941d]fnic_isr_msix_wq_copy@<None>#<None>+0x18 stack: 0x417f0000001f
    YYYY-MM-DDT15:12:27.897Z cpu28:4124)0x412240707b28:[0x41801bea2c90][email protected]#9.2+0x2b

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 　

Cause

This issue can occur if there is a storage outage in a Cisco environment with an fnic driver earlier than version 1.5.0.8. The fnic driver returns a device reset without freeing all the resources assigned to that device. When the fnic driver starts to tear down the LUN information, it encounters a stale pointer, which leads to a purple diagnostic screen.

Resolution

This issue is resolved in Cisco fnic FC driver version 1.6.0.17a and later. Available at VMware Downloads.

To check your fnic driver version installed on the ESXi:

1. Connect to the ESXi host via SSH.
2. Run this command:
   
   esxcfg-info | less
   
   
3. To search for the driver, run this command:
   
   /vmkmod/fnic
   
   
4. Press Enter and search for this entry:
   
   ”Version..."
   
   
   Note: Alternatively, run this command to see the current fnic version:
   
   vmkload_mod -s fnic
   

If the driver version is 1.6.0.17a or later, continue diagnostic analysis.

If the driver version is earlier than 1.6.0.17a, download the latest version of the driver from VMware, and then install it on your host(s). For more information, see Installing async drivers on VMware ESXi 5.0, 5.1, and 5.5(2005205).

You can download a Cisco fnic driver from VMware Downloads:

• Download VMware ESXi 5.0 Driver for Cisco fnic
• Download VMware 5.0/5.1 Driver for Cisco fnic

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.
How to download and install async drivers in ESXi 5.x/6.x
使用 Cisco fnic 驱动程序的 ESXi 5.x 主机遇到紫色诊断屏幕，并显示异常 14 和 fnic 驱动程序错误