PSOD: "PF Exception 14 in world ####:amsd"
Article ID: 454359
Updated On:
Products
Issue/Introduction
- A PSOD occurs on an ESXi host showing a " #PF Exception 14 in world #######:amsd IP 0x########## addr ox###".
- The host crash was observed after updating the qlnativefc driver firmware.
- The PSOD back trace will contain modules similar to the following:
qlnativefcAsyncGpnftGnnftSpDone@(qlnativefc)#<None>+0x31c
qlnativefcAbortAllCmds@(qlnativefc)#<None>+0xab
qlnativefcAbortIspCleanup@(qlnativefc)#<None>+0x31f
qlnativefcAbortIsp@(qlnativefc)#<None>+0x305
qlnativefcMailboxCommand@(qlnativefc)#<None>+0x12d4
qlnativefcGetFirmwareState@(qlnativefc)#<None>+0x6e
qlnativefcGetHbaPort@(qlnativefc)#<None>+0x357
[email protected]#v2_13_0_0+0x149
[email protected]#v2_13_0_0+0x108
[email protected]#v2_13_0_0+0x1b9
VMKAPICharDevDevfsWrapIoctl@vmkernel#nover+0x73
CharDriverIoctl@vmkernel#nover+0x76
DevFSIoctl@vmkernel#nover+0x5d3
FSSVec_Ioctl@vmkernel#nover+0x1c
FSSObjectIoctlCommon@vmkernel#nover+0x1f0
FSS_IoctlByFH@vmkernel#nover+0xa4
UserFile_PassthroughIoctl@vmkernel#nover+0x23
UserVmfs_Ioctl@vmkernel#nover+0x28- In the /var/run/log/vmkernel.log file of the affected ESXi host, the following entries may be observed:
- Firmware commands are shown to timeout:
Wa(180) vmkwarning: cpu23:2099447)WARNING: qlnativefc: vmhba3(c3:0.1): qlnativefcMailboxCommand:405:Mailbox command 0x69 timeout occurred. Issuing ISP abort. In(182) vmkernel: cpu23:2099447)qlnativefc: vmhba3(c3:0.1): qla27xxFwdump:1243:Attempting to take firmware dump Wa(180) vmkwarning: cpu16:2099447)WARNING: qlnativefc: vmhba3(c3:0.1): qlnativefcMailboxCommand:405:Mailbox command 0x69 timeout occurred. Issuing ISP abort. In(182) vmkernel: cpu16:2099447)qlnativefc: vmhba3(c3:0.1): qla27xxFwdump:1243:Attempting to take firmware dump Wa(180) vmkwarning: cpu16:2099447)WARNING: qlnativefc: vmhba3(c3:0.1): qlnativefcMailboxCommand:405:Mailbox command 0x69 timeout occurred. Issuing ISP abort. - Recovery attempts by the driver are shown to fail:
In(182) vmkernel: cpu5:2101991)qlnativefc: vmhba3(c3:0.1): qlnativefcAbortIspCleanup:6004:Performing ISP error recovery - ha= 0x43162201c000. In(182) vmkernel: cpu18:2099447)qlnativefc: vmhba3(c3:0.1): qlnativefcAbortIspCleanup:6004:Performing ISP error recovery - ha= 0x43162201c000. In(182) vmkernel: cpu0:2101991)qlnativefc: vmhba3(c3:0.1): qlnativefcAbortIspCleanup:6004:Performing ISP error recovery - ha= 0x43162201c000.
- Firmware commands are shown to timeout:
Environment
ESXi 8.x
Cause
A timing conflict occurred between two separate processing threads within the driver. One thread attempted to retrieve attributes for a Fibre Channel port (qlnfcEdifSadbGetSaIndex) while a parallel thread was actively removing sessions. This caused the first thread to access an empty memory pointer (0x0), resulting in a critical Page Fault (#PF) and an immediate system panic.
This race condition was directly exposed by a severe loop of HBA mailbox timeouts (Mailbox command 0x69 timeout occurred). Immediately following the firmware upgrade, the adapter (vmhba3) began timing out repeatedly. This forced the driver into a constant loop of resetting the adapter and collecting firmware dumps, creating the volatile processing environment where the timing race condition could manifest.
Resolution
Please contact the hardware/hba vendor for further assistance.
Feedback
