PSOD on ESXi host with error "@BlueScreen: vmklinux: Attempted double completion"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The ESXi 5.x or 6.x host fails with a purple diagnostic screen.
• You see stack entries similar to:
  
  
  cpu9:34855)@BlueScreen: vmklinux: Attempted double completion
  
  
  cpu9:34855)Code start: 0x418017c00000 VMK uptime: 1:10:40:17.503
  cpu9:34855)0x4123a09dd4c0:[0x418017c8d429]PanicvPanicInt@vmkernel#nover+0x575 stack: 0x412e81d26200
  cpu9:34855)0x4123a09dd4e0:[0x418017c8d579]Panic_vPanic@vmkernel#nover+0x31 stack: 0x412300000010
  cpu9:34855)0x4123a09dd540:[0x418017cb3afb]vmk_PanicWithModuleID@vmkernel#nover+0x3b stack: 0x0
  cpu9:34855)0x4123a09dd5d0:[0x4180182fce64][email protected]#9.2+0x3f4 stack: 0x4123a09dd61
  cpu9:34855)0x4123a09dd8f0:[0x418018c0a082]complete_scsi_command@<None>#<None>+0x16a stack: 0x4108666a2a20
  cpu9:34855)0x4123a09dd940:[0x418018c0c932]do_hpsa_intr_msi@<None>#<None>+0xf2 stack: 0x412300001050
  cpu9:34855)0x4123a09dd980:[0x4180182c2d7e][email protected]#9.2+0x2a stack: 0x4123a09dd9c0
  cpu9:34855)0x4123a09dda10:[0x418017c6b206]IRQ_DoInterrupt@vmkernel#nover+0x33e stack: 0x4123a09dda70
  cpu9:34855)0x4123a09dda50:[0x418017c643f3]IDT_IntrHandler@vmkernel#nover+0x12b stack: 0x4123a09ddb70
  cpu9:34855)0x4123a09dda60:[0x418017cf2064]gate_entry@vmkernel#nover+0x64 stack: 0x0
  cpu9:34855)0x4123a09ddb70:[0x418017faa73a]Power_HaltPCPU@vmkernel#nover+0x1fe stack: 0x0
  cpu9:34855)0x4123a09ddbe0:[0x418017e543dc]CpuSchedIdleLoopInt@vmkernel#nover+0x550 stack: 0x4108711eee10
  cpu9:34855)0x4123a09ddd30:[0x418017e5a0b2]CpuSchedDispatch@vmkernel#nover+0x17be stack: 0x4123a09ddd70
  cpu9:34855)0x4123a09ddda0:[0x418017e5b20b]CpuSchedWait@vmkernel#nover+0x237 stack: 0x410800000001
  cpu9:34855)0x4123a09dde40:[0x418017cde81e]WorldWaitInt@vmkernel#nover+0x2c6 stack: 0x4123a09dde80
  cpu9:34855)0x4123a09dde70:[0x4180181a3355]UserThread_Sleep@<None>#<None>+0x55 stack: 0x2faf08000000000
  cpu9:34855)0x4123a09ddeb0:[0x4180181b488c]LinuxThread_Nanosleep@<None>#<None>+0x88 stack: 0x0
  cpu9:34855)0x4123a09ddf00:[0x418018186080]User_LinuxSyscallHandler@<None>#<None>+0x3f4 stack: 0x4123a09ddf20
  cpu9:34855)0x4123a09ddf10:[0x418017caae7d]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0xfff9bf08
  cpu9:34855)0x4123a09ddf20:[0x418017cf2064]gate_entry@vmkernel#nover+0x64 stack: 0x

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5

Cause

This issue occurs when an abort request arrives just before the controller completes the associated IO request. In this case, the abort handler and the normal completion code attempt to complete the I/O resulting in PSOD.

Resolution

To resolve this issue, the HPE Smart Array (HPSA) Driver needs to be upgraded to the .120-1 version. This HPE driver version has been improved to allow better coordination between abort handler and completion path so that only the completion code performs the actions necessary to complete an I/O.

The download link for the drivers are mentioned in the HPE advisory.