ESXi hosts experience PSOD with smartpqi driver

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts utilizing the smartpqi driver may experience a Purple Screen of Death (PSOD).

• The system crash occurs unexpectedly, and the following stack trace is observed:
  
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)cr0=0x8001003d cr2=0x7f9cf3f7f480 cr3=0xa0179000 cr4=0x10216c
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)FMS=06/55/7 uCode=0x5003103
  *PCPU10:529744/LSI-529728:0
  PCPU  0: SSVSSSSSVSSSVSSS
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)Code start: 0x420031000000 VMK uptime: 1039:05:47:42.725
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)Saved backtrace from: pcpu 10 Heartbeat NMI
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9ac10:[0x420031e5b6b3]pqisrc_build_send_io@(smartpqi)#<None>+0x2e4 stack: 0x100000001
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9ad30:[0x420031e4357c]pqisrc_scsi_queue_command@(smartpqi)#<None>+0x2d9 stack: 0x43037b0926c0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9ad80:[0x42003140ae78]SCSIIssueCommandDirect@vmkernel#nover+0xf9 stack: 0x210031
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9adf0:[0x42003140c578]SCSIStartAdapterCommands@vmkernel#nover+0x2fd stack: 0x7b087d10
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9ae70:[0x42003141eccf]SCSIStartPathCommands@vmkernel#nover+0x4b0 stack: 0x43037b0926c0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9afa0:[0x420031424d47]SCSIIssueAsyncPathCommandDirect@vmkernel#nover+0x2d4 stack: 0x0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b0b0:[0x420031426c37]vmk_ScsiIssueAsyncPathCommandDirect@vmkernel#nover+0x1c stack: 0x4558c20bff40
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b0d0:[0x420031d00bd1]HppIssueCommand@(hpp)#<None>+0xfa stack: 0x0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b160:[0x420031d011c5]HppDeviceStartCommandProcessing@(hpp)#<None>+0x18a stack: 0x4559309ef580
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b5d0:[0x4200313f2812]SCSIStartDeviceInt@vmkernel#nover+0x53 stack: 0x4200313f280a
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b610:[0x4200313fa07d]SCSIDirectSubmitOrQueueCommand@vmkernel#nover+0x18e stack: 0x0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b670:[0x4200313fa88d]SCSIAsyncDeviceCommand@vmkernel#nover+0x252 stack: 0x109
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b740:[0x420031050da6]FDS_CommonAsyncIO@vmkernel#nover+0x43 stack: 0x43037b0eb420
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b780:[0x42003104bf1e]FDS_AsyncIO@vmkernel#nover+0x6a3 stack: 0x455930fee018
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b820:[0x42003200e5d7]Fil6_HandleReadIO@esx#nover+0x128 stack: 0x100000
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b880:[0x420032010bf2]Fil6_FileIOInt@esx#nover+0xe73 stack: 0x430ecf14ede0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9b9f0:[0x420032011fa0]Fil6_FileIOIntWithRetry@esx#nover+0x55 stack: 0x420031150a18
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9ba90:[0x4200320122fb]Fil6_FileIO@esx#nover+0x138 stack: 0x420042805690
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bb90:[0x42003103b2d7]FSSVec_FileIO@vmkernel#nover+0x20 stack: 0x43044cb9f150
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bbb0:[0x420031038b08]FSSFileIO@vmkernel#nover+0x1dd stack: 0x45388a221000
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bc10:[0x420031038c79]FSS_AsyncFileIO@vmkernel#nover+0xe stack: 0x430637c18b40
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bc30:[0x4200314f0007]VSCSI_FSCommand@vmkernel#nover+0xb98 stack: 0x3b3a09
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bd10:[0x4200314e3ad4]VSCSI_IssueCommandBE@vmkernel#nover+0x35 stack: 0x453893b9bdb0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bd50:[0x4200314e4a12]VSCSIExecuteCommandInt@vmkernel#nover+0x4c3 stack: 0xa000000000
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)0x453893b9bdd0:[0x4200314e5e98]VSCSI_VmkExecuteCommand@vmkernel#nover+0x71 stack: 0x100000000
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)base fs=0x0 gs=0x420042800000 Kgs=0x0
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)1 other PCPU is in panic.
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)NMI: 690: NMI IPI: RIPOFF(base):RBP:CS [0xe5b6b4(0x420031000000):0x43090406e960:0xf48] (Src 0x1, CPU10)
  YYYY-MM-DDTHH:MM:SS.SSSZ cpu10:529744)NMI: 690: NMI IPI: RIPOFF(base):RBP:CS [0xe5b6b4(0x420031000000):0x43090406e960:0xf48] (Src 0x1, CPU10)

Environment

VMware ESXi 7.0

Cause

This issue is caused by a specific behavior within the OEM smartpqi driver provided by the hardware vendor.

Resolution

Update both the smartpqi driver and firmware to the latest versions listed in the VMware by Broadcom Compatibility Guide.

Additional Information

For instructions on how to download ESXi drivers, please refer to Finding ESXi Drivers in the Broadcom Support Portal.