Esxi 8.0 U3 host carshes with PSOD due to qlnativefc race condition
search cancel

Esxi 8.0 U3 host carshes with PSOD due to qlnativefc race condition

book

Article ID: 451899

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Esxi 8.0 U3 crashes with below backtrace

#0  0x0000420012d1742c in qlnativefcAsyncGpnftGnnftSpDone () from /opt/nfs/vcf_gs_csp_prd_users/mw035774/build/drivers/qlnativefc-5.4.85.0/usr/lib/vmware/vmkmod/qlnativefc
#1  0x0000420012d5851a in qla24xxElsCtEntry () from /opt/nfs/vcf_gs_csp_prd_users/mw035774/build/drivers/qlnativefc-5.4.85.0/usr/lib/vmware/vmkmod/qlnativefc
#2  0x0000420012d5902e in qla24xxProcessResponseQueue () from /opt/nfs/vcf_gs_csp_prd_users/mw035774/build/drivers/qlnativefc-5.4.85.0/usr/lib/vmware/vmkmod/qlnativefc
#3  0x0000420012d60d56 in qla24xxMsixRspQ () from /opt/nfs/vcf_gs_csp_prd_users/mw035774/build/drivers/qlnativefc-5.4.85.0/usr/lib/vmware/vmkmod/qlnativefc
#4  0x0000420011d6003c in IntrCookieCallHandler (cookie=27, clientData=0x4313d301da50, handler=0x430286caac10) at bora/vmkernel/main/intrCookie.c:2026
#5  IntrCookieBH (unused=<optimized out>) at bora/vmkernel/main/intrCookie.c:2204
#6  0x0000420011d3ff8d in BHCallHandlers (pending=1) at bora/vmkernel/main/bh.c:761
#7  BH_DrainAndDisableInterrupts (canReschedule=canReschedule@entry=1 '\001') at bora/vmkernel/main/bh.c:1060
#8  0x0000420011d61d00 in IntrCookie_VmkernelInterrupt (vector=112, vectorData=vectorData@entry=0, fullFrame=fullFrame@entry=0x4539e3f9b920) at bora/vmkernel/main/intrCookie.c:4223
#9  0x00004200122a990b in IDTHandleInterrupt (fullFrame=0x4539e3f9b920) at bora/vmkernel/main/x86/idt.c:1674
#10 IDT_IntrHandler (fullFrame=0x4539e3f9b920) at bora/vmkernel/main/x86/idt.c:1697
#11 0x00004200122a20c7 in gate_entry () at bora/vmkernel/main/x86/gates64.S:175
#12 0x0000420011c90f00 in CPU_StiMwaitInstr (hints=0, extensions=0) at bora/vmkernel/hardware/x86/cpu_int_arch.h:118

  • /var/log/vmware/vmkernel.log reports below errors 

YYY-MM-DD:45.814Z cpu24:6759639)qlnativefc: vmhba0(af:0.0): qlnativefcAsyncEvent:1135:Discard RND Frame -- fffe 0001 0148.
YYYY-MM-DD:45.831Z cpu40:2099387)Tcpip_Vmk: 123: INFO: sosetopt set SO_AUTOBIND
YYYY-MM-DD:46.096Z cpu60:2097676)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:1927:(0:57) Dropped frame(s) detected (798720 of 798720 bytes).
YYYY-MM-DD:46.096Z cpu6:9631561)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:1927:(0:16) Dropped frame(s) detected (524288 of 524288 bytes).
YYYY-MM-DD:46.096Z cpu60:2097676)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:2076:C0:T0:L57 - FCP command status: 0x15-0x0 (0x2) portid=c82620 oxid=0x20e cdb=28000a len=798720 rspInfo=0x0 resid=0x0 fwResid=0xc3000 host status = 0x2 device st$
YYYY-MM-DD:46.096Z cpu6:9631561)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:2076:C0:T0:L16 - FCP command status: 0x15-0x0 (0x2) portid=c82620 oxid=0x20c cdb=28000b len=524288 rspInfo=0x0 resid=0x0 fwResid=0x80000 host status = 0x2 device sta$
YYYY-MM-DD:46.096Z cpu6:9631561)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:1927:(0:16) Dropped frame(s) detected (524288 of 524288 bytes).
YYYY-MM-DD:46.096Z cpu6:9631561)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:2076:C0:T0:L16 - FCP command status: 0x15-0x0 (0x2) portid=c82620 oxid=0x20d cdb=28000b len=524288 rspInfo=0x0 resid=0x0 fwResid=0x80000 host status = 0x2 device sta$
YYYY-MM-DD:46.096Z cpu78:2098250)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x28 (0x45ea185cc3c0, 5078618) to dev "naa.60060e8008827c000050827c000050f9" on path "vmhba0:C0:T0:L57" Failed:
YYYY-MM-DD:46.096Z cpu6:9631561)qlnativefc: vmhba0(af:0.0): qlnativefcStatusEntry:1927:(0:16) Dropped frame(s) detected (524288 of 524288 bytes).
YYYY-MM-DD:46.096Z cpu78:2098250)NMP: nmp_ThrottleLogForDevice:3898: H:0x2 D:0x0 P:0x0 . Act:EVAL. 

Environment

Esx 8.0 U3

Cause

This issue is caused by a race condition within the qlnativefc driver (between version: 2099344 and 2098071) while the driver was recovering from an error state.
During this recovery process, a qlnativefc thread attempted to access an fcport for a vmhba that was NULL, which resulted in vmhba0 reporting numerous errors at the time of the PSOD.

Resolution

To fix this issue upgrade the drivers and firmware to the latest version. 

Powered by Wolken Software