HPE Servers running ESXi 6.x version and configured with an HPE FlexFabric 20Gb 2-port 650FLB adapter, may log "Wake NOT Set" messages in the vmkernel log and eventually a PSOD (purple screen of death) may occur

Products

• VMware vSphere ESXi

Issue/Introduction

To avoid PSOD (purple screen of death).

Symptoms:

• ESXi host Crashed with PSOD (purple screen of death) caused by brcmfcoe driver referencing __lpfc_sli_get_iocbq.
• Any of the following HPE servers running VMware ESXi 6.0, VMware ESXi 6.5 or VMware ESXi 6.7 and configured with an HPE FlexFabric 20Gb 2-port 650FLB Adapter with driver version 12.0.1211.0 (or prior):
  
      HPE ProLiant BL460c Gen10 Server Blade
      HPE ProLiant BL460c Gen9 Server Blade
      HPE ProLiant BL660c Gen9 Server
• Host crashes with or without load.
• The host may crash with a wake NOT set" message is logged in the vmkernel logs as below

2019-01-20T11:36:22.861Z cpu7:32854)elxnet: elxnet_startQueue:5365: [vmnic5] RxQ, QueueIDVal:4
2019-01-20T11:37:07.966Z cpu19:31782666)WARNING: brcmfcoe: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x0
2019-01-20T11:37:07.969Z cpu19:31782666)World: 9762: PRDA 0x418044c00000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b
2019-01-20T11:37:07.969Z cpu19:31782666)World: 9764: TR 0x4020 GDT 0x4394b8521000 (0x402f) IDT 0x4180244ca000 (0xfff)
2019-01-20T11:37:07.969Z cpu19:31782666)World: 9765: CR0 0x80010031 CR3 0x20cc2f9000 CR4 0x42768
2019-01-20T11:37:08.036Z cpu19:31782666)Backtrace for current CPU #19, worldID=31782666, rbp=0x4305405fae70
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8a8:[0x418024ca1f69]__lpfc_sli_get_iocbq@<None>#<None>+0x51 stack: 0x4305407ef330, 0x430
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8b0:[0x418024ca27cd]lpfc_sli_get_iocbq@<None>#<None>+0x25 stack: 0x4305405fae70, 0x41802
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851b8d0:[0x418024c813cb]lpfc_mgmt_send_ct@<None>#<None>+0x98b stack: 0x0, 0x439400000028, 0x
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851ba20:[0x418024af3b6f][email protected]#v2_3_0_0+0x13f stack: 0x1,
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bae0:[0x418024af4a26][email protected]#v2_3_0_0+0x5ea stack: 0x4305407ef
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bc70:[0x418024495130]VMKAPICharDevDevfsWrapIoctl@vmkernel#nover+0x80 stack: 0x43940000001
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bce0:[0x418024697694]DevFSIoctl@vmkernel#nover+0x684 stack: 0x43132dd9cc70, 0xbcd00000000
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bdb0:[0x418024673d38]FSSObjectIoctlCommon@vmkernel#nover+0x9c stack: 0x4394b851be43, 0x43
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be00:[0x41802467743c]FSS_IoctlByFH@vmkernel#nover+0xbc stack: 0x1f08d7b0, 0x200, 0x1f08d7
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be40:[0x4180249e8d4b]UserFile_PassthroughIoctl@<None>#<None>+0x3f stack: 0x43132dda2be8,
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851be60:[0x418024a30764]UserVmfs_Ioctl@<None>#<None>+0x24 stack: 0x43132dd9c0e0, 0x4180249cd
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bec0:[0x4180249f3074]LinuxFileDesc_Ioctl@<None>#<None>+0x5c stack: 0x4394b851bf30, 0x36,
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bef0:[0x4180249c8663]User_LinuxSyscallHandler@<None>#<None>+0xd7 stack: 0x0, 0x0, 0x0, 0x
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bf20:[0x41802448ee91]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0x0, 0x13b, 0x0,
2019-01-20T11:37:08.036Z cpu19:31782666)0x4394b851bf30:[0x4180244c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x36, 0x200, 0x1f08d7b0,
2019-01-20T11:37:08.068Z cpu19:31782666)VMware ESXi 6.0.0 [Releasebuild-10474991 x86_64]
#PF Exception 14 in world 31782666:ams-main IP 0x418024ca1f69 addr 0xe0


Or a similar coredump to:

2019-01-26T12:18:58.687Z cpu22:35927)WARNING: brcmfcoe: lpfc_sli_issue_iocb_wait:10765: 0:0330 IOCB wake NOT set, Data x24 x0
2019-01-26T12:18:58.697Z cpu2:33532)World: 9762: PRDA 0x418040800000 ss 0x0 ds 0x10b es 0x10b fs 0x0 gs 0x13b
2019-01-26T12:18:58.697Z cpu2:33532)World: 9764: TR 0x4020 GDT 0x43929c721000 (0x402f) IDT 0x4180008ca000 (0xfff)
2019-01-26T12:18:58.697Z cpu2:33532)World: 9765: CR0 0x80010031 CR3 0x20fa85f000 CR4 0x42768
2019-01-26T12:18:58.762Z cpu2:33532)Backtrace for current CPU #2, worldID=33532, rbp=0x43051d9d0010
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b8e8:[0x4180010a1f69]__lpfc_sli_get_iocbq@<None>#<None>+0x51 stack: 0x43051d91de70, 0x430
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b8f0:[0x4180010a27cd]lpfc_sli_get_iocbq@<None>#<None>+0x25 stack: 0x43051d9d0010, 0x41800
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b910:[0x4180010acb24]lpfc_sli4_handle_eqe@<None>#<None>+0x6ac stack: 0x430336d7a140, 0xa0
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1b9d0:[0x4180010ad6e1]lpfc_sli4_intr_bh_handler@<None>#<None>+0x89 stack: 0x28, 0x41000b11
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1ba00:[0x418000859199]IntrCookieBH@vmkernel#nover+0x299 stack: 0x0, 0x43051d95bca0, 0x4391
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1baa0:[0x41800083329e]BH_DrainAndDisableInterrupts@vmkernel#nover+0xe2 stack: 0x1010000000
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bb30:[0x4180008571b2]IDT_IntrHandler@vmkernel#nover+0x1ce stack: 0x0, 0x0, 0x0, 0x4180008
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bb60:[0x4180008c8067]gate_entry_@vmkernel#nover+0x0 stack: 0x0, 0x0, 0x0, 0x0, 0x41804080
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bc20:[0x418000b067fa]Power_HaltPCPU@vmkernel#nover+0x1ee stack: 0x417fc0a84f60, 0x4180409
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bc70:[0x418000a12dc8]CpuSchedIdleLoopInt@vmkernel#nover+0x2f8 stack: 0x39266d1018ad98, 0x
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bcf0:[0x418000a16522]CpuSchedDispatch@vmkernel#nover+0x15fe stack: 0x439117e27100, 0x4391
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1be10:[0x418000a17104]CpuSchedWait@vmkernel#nover+0x240 stack: 0x0, 0x430769ee8080, 0x5d01
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1be90:[0x418000a17445]CpuSchedTimedWaitInt@vmkernel#nover+0xc9 stack: 0x430700002001, 0x41
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bf10:[0x418000a17516]CpuSched_TimedWait@vmkernel#nover+0x36 stack: 0x430769ee8080, 0x4180
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bf30:[0x418000850a5d]helpFunc@vmkernel#nover+0x611 stack: 0x0, 0x430769e11b60, 0x33, 0x0,
2019-01-26T12:18:58.762Z cpu2:33532)0x439117e1bfd0:[0x418000a17d9e]CpuSched_StartWorld@vmkernel#nover+0xa2 stack: 0x0, 0x0, 0x0, 0x0, 0
2019-01-26T12:18:58.795Z cpu2:33532)VMware ESXi 6.0.0 [Releasebuild-10474991 x86_64]
#PF Exception 14 in world 33532:helper51-3 IP 0x4180010a1f69 addr 0xe0
PTEs:0x160bba027;0x20f9fb1027;0x0;

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Cause

The completion handler associated with the IOCB was never called.

Resolution

To resolve the issue, upgrade the brcmfcoe driver:

• For ESXi 6.0, use brcmfcoe driver version 12.0.1110.39 
• For ESXi 6.5 , use  brcmfcoe driver version 12.0.1216.4 
• For ESXi 6.7, use brcmfcoe driver version 12.0.1216.4

Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends or accepts any responsibility for the content of such sites 

IMPORTANT: Refer to HPE customer advisory  for more information

Workaround:
No  workaround.

Additional Information

Procedure to install driver using various methods refer to VMware KB : How to download and install async drivers in ESXi 5.x/6.x

Impact/Risks:
Restart  of ESXi host is required after driver is installed.