PSOD (purple screen of death) occurs with a backtrace Timeout with only XX of XX PCPUS having completed pre-stop callback

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
You will see the backtraces similar to the following in PSOD (purple screen of death) screen.

0x451b17c1bc80:[0x42000f0fc6ce]PanicvPanicInt@vmkernel#nover+0x2c6 stack: 0x42000f0fc6ce, 0x0, 0x451b17c1bc80, 0x42000f0fcbad, 0x42000f0fcbad
0x451b17c1bd30:[0x42000f0fcbac]Panic_NoSave@vmkernel#nover+0x4d stack: 0x451b17c1bd90, 0x451b17c1bd50, 0x27, 0x26, 0x27
0x451b17c1bd90:[0x42000f07a607]SMPWaitForPCPUs@vmkernel#nover+0x54 stack: 0x1, 0x451b17c1be70, 0x27, 0x42000f07bea8, 0x0
0x451b17c1bdc0:[0x42000f07bea7]SMP_SynchronizedCallFunction@vmkernel#nover+0x148 stack: 0x42000f033db4, 0x0, 0x0, 0x100420080000000, 0xffffff7fff
0x451b17c1beb0:[0x42000f0340e1]NRandomHwrngFillPool@vmkernel#nover+0x2e6 stack: 0x431c65000000, 0x4000, 0x3, 0x431c65000000, 0x10
0x451b17c1bfe0:[0x42000f35ef79]CpuSched_StartWorld@vmkernel#nover+0x82 stack: 0x0, 0x42000f0c5304, 0x0, 0x0, 0x0
0x451b17c1c000:[0x42000f0c5303]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 7.0.x

Cause

Timer cancellation has to be issued with preemption enabled. However, preemption is disabled when it should be enabled to cancel the timer due to the bug. As the result, NMI watchdog detects the timer that keeps running too long, which cause the PSOD.

Resolution

This issue is resolved in ESXi 7.0 Update 1, available at  vSphere ESXi 7.0 U1