ESXi reports PSOD for a rare race condition of double free when using vVol

Products

• VMware vSphere ESXi

Issue/Introduction

PSOD occurred on the ESXi host with the following backtrace, caused by vVOL

Panic Message: @BlueScreen: PANIC bora/vmkernel/main/dlmalloc.c:4944 - Usage error in dlmalloc
Backtrace:
  0x453996a9b940:[0x42001e77bbc0]PanicvPanicInt@vmkernel#nover+0x20c stack: 0x431aaec16000, 0x42001e77bbc0, 0x0, 0x420000000001, 0x42001e77bbc0
  0x453996a9b9f0:[0x42001e77c33c]Panic_NoSave@vmkernel#nover+0x4d stack: 0x453996a9ba50, 0x453996a9ba10, 0x42001ed399a1, 0x42001ecf3383, 0x1350
  0x453996a9ba50:[0x42001e7326f3]DLM_free@vmkernel#nover+0x15c stack: 0x4335f440ce60, 0x42001e73b1ee, 0x453996a9bd78, 0x420000000000, 0x4309fdbdd900
  0x453996a9ba70:[0x42001e73b1ed]Heap_Free@vmkernel#nover+0xb2 stack: 0x4309fdbdd900, 0x430ec3e694b0, 0x4335f441d7c0, 0x430ec3e694b0, 0x430ec3e694b0
  0x453996a9bac0:[0x4200203247ee]UpdateCachedPolicyInVVolDevice@(vvol)#<None>+0x57 stack: 0x80000000, 0x42002032a471, 0x430cb9e3a420, 0xc2d, 0x0
  0x453996a9bae0:[0x42002032a470]VVolIoctl@(vvol)#<None>+0x7dd stack: 0x0, 0x430cb9e3a420, 0x4c3618e0, 0x42001e642678, 0xc2d
  0x453996a9bb70:[0x42001e655d76]DevFSIoctl@vmkernel#nover+0xfa7 stack: 0x0, 0x430cb9e3a580, 0x689eab3b, 0x430cb9e3a420, 0x430a74614a10
  0x453996a9bc30:[0x42001e642677]FSSVec_Ioctl@vmkernel#nover+0x1c stack: 0x5a3831322e37323a, 0x313a343475706320, 0x2939353436323035, 0x32203a4953435356, 0x689eab3b
  0x453996a9bc40:[0x42001ecbb6d1]FSSObjectIoctlCommon@vmkernel#nover+0x222 stack: 0x2939353436323035, 0x32203a4953435356, 0x689eab3b, 0x383335340d0081f3, 0x3832343434383930
  0x453996a9bd10:[0x42001ecbb49f]FSS_IoctlByFH@vmkernel#nover+0xa4 stack: 0x453996a9bd62, 0x430ec3d0d7c0, 0x4c3618e0, 0x0, 0x430ec3e694b0
  0x453996a9bd60:[0x42001ec49106]VSCSI_FSSetSchedPolicy@vmkernel#nover+0x5f stack: 0x12c000000000, 0x10000, 0x100000000000000, 0x81f3689eab3b0000, 0x81f3689eab3b0d00
  0x453996a9be00:[0x42001ec69353]VSCSI_SetSchedPolicy@vmkernel#nover+0x438 stack: 0xffffffffffffffff, 0x100000001, 0x453996a9be4c, 0x0, 0x1fff64337d184c
  0x453996a9be70:[0x42001ec2c8b5]UWVMKSyscall_SetSchedPolicy@vmkernel#nover+0x6a stack: 0x0, 0xe5490f0040380c, 0x0, 0x42001ec35af3, 0xbad9a282e0
  0x453996a9bea0:[0x42001ec35af2]UW64VMKSyscallUnpackSetSchedPolicy@vmkernel#nover+0x73 stack: 0xffffffffffffffff, 0xffffffff, 0x431a6d202010, 0x0, 0x431a6d21b810
  0x453996a9bef0:[0x42001ebd06bb]User_UWVMK64SyscallHandler@vmkernel#nover+0x104 stack: 0x42001ebcfd04, 0x0, 0x0, 0x0, 0x0
  0x453996a9bf40:[0x42001ec98105]SyscallUWVMK64@vmkernel#nover+0xc1 stack: 0x0, 0x0, 0xbad62d5ce1, 0xbad63df500, 0xffffffffffffffff

Environment

VMware vSphere ESXi 8.x

Cause

The VVol device scheduling policy is modified without acquiring a lock, which can result in two or more threads freeing it simultaneously, potentially leading to a PSOD

Resolution

This issue occurs only under rare conditions and has been resolved in the ESXi 8.0.3 P07 release.