Multiple ESXi Hosts in the cluster failed with a PSOD during VM Quiescing

Products

• VMware vSphere ESXi

Issue/Introduction

Multiple ESXi hosts running version 8.0 Update 3f fail with a Purple Diagnostic Screen (PSOD) during Virtual Machine quiescing or stunning operation :

 

cpu86:107944396)@BlueScreen: #PF Exception 14 in world #########:vmx IP ############## addr 0x0
PTEs:0x321d944027;0x6010434027;0x0;
cpu86:107944396)Code start: 0x420013400000 VMK uptime: 118:22:29:11.756
cpu86:107944396)0x45397361b4f8:[0x4200135d83c8]PFrame_GetPinCount@vmkernel#nover+0x0 stack: 0x431a7f20cd70
cpu86:107944396)0x45397361b500:[0x420013634faa]VmMemPin_DecCount@vmkernel#nover+0x1b stack: 0x40000e02e4
cpu86:107944396)0x45397361b530:[0x420013635109]VmMemPinUnpinPage@vmkernel#nover+0x8a stack: 0x2
cpu86:107944396)0x45397361b570:[0x42001363603c]VmMemPin_ReleasePhysMemRange@vmkernel#nover+0x75 stack: 0xb03b8
cpu86:107944396)0x45397361b5d0:[0x4200135dc08c]PhysMemReleaseMapping@vmkernel#nover+0x139 stack: 0x45396671f000
cpu86:107944396)0x45397361b600:[0x4200135dd5c3]PhysMem_DestroyToken@vmkernel#nover+0x18 stack: 0x0
cpu86:107944396)0x45397361b620:[0x4200136a764a]Vmxnet3VMKDevUnmapTQs@vmkernel#nover+0x43 stack: 0x4305fa4af300
cpu86:107944396)0x45397361b650:[0x4200136a9a44]Vmxnet3VMKDevCleanupClientData@vmkernel#nover+0x171 stack: 0x4305fa4af300
cpu86:107944396)0x45397361b6a0:[0x4200136be294]Vmxnet3VMKDevCleanup@vmkernel#nover+0xf5 stack: 0x430568d24740
cpu86:107944396)0x45397361b6d0:[0x42001366d906]PortCleanupBeforeDisable@vmkernel#nover+0x3a3 stack: 0x430199c70086
cpu86:107944396)0x45397361b740:[0x42001366e456]PortDoDisable@vmkernel#nover+0x5b stack: 0x0
cpu86:107944396)0x45397361b770:[0x42001366e545]Port_QuiesceAndDisable@vmkernel#nover+0x2a stack: 0x6000118
cpu86:107944396)0x45397361b7a0:[0x4200136c3fab]Vmxnet3VMKDev_Enable@vmkernel#nover+0xc8c stack: 0x0
cpu86:107944396)0x45397361b870:[0x420013a3c7e0]UW64VMKPrivateSyscallUnpackVmxnet3Enable@vmkernel#nover+0xd1 stack: 0x0
cpu86:107944396)0x45397361bef0:[0x4200139d3c2b]User_UWVMK64SyscallHandler@vmkernel#nover+0x104 stack: 0x4200139d3274
cpu86:107944396)0x45397361bf40:[0x420013a9d445]SyscallUWVMK64@vmkernel#nover+0xc1 stack: 0x0
cpu86:107944396)base fs=0x0 gs=0x420055800000 Kgs=0x0

 

Environment

ESXi 8.0 Update 3f

Cause

FSR(Fast Suspend-Resume) a non-disruptive technique used in vSphere to apply live patches, hot-add hardware (vCPU, memory), and for Storage vMotion ...etc. During this process the VMs are Quiesced/Stunned as required. As part of this VMXNet3 should also be Quiesced and the backend port deleted/released. However, due to WOL feature (Wake up on LAN) being enabled, the port on the host gets enabled immediately. VMXNet3 is not being quiesced when the VM is stunned. VMXNet3 device state gets into a unexpected state causing the PSOD

Resolution

Workaround  is to disable WOL at the host and VM level.