ESXi 7.0 and 8.0 host fails with a purple diagnostic screen that includes "NUMASched_PageNum2PhysicalDomain" and "vMotionRecv_ReadCompletePages" in the backtrace

Products

• VMware vSphere ESXi

Issue/Introduction

This article describes a specific issue. If you experience all of the above symptoms, consult the sections below. If you experience some but not all of these symptoms, your issue is not related to this article. Search for your symptoms in the Knowledge Base or file a Support Request.

Symptoms:

• An ESXi 7.0 and 8.0 host stops responding and displays a purple diagnostic screen indicating "NMI IPI: Panic requested by another PCPU. RIPOFF(base)"
• The backtrace includes entries similar to the following:

NUMASched_PageNum2PhysicalDomain@vmkernel#nover+0x52 stack:
NUMASched_UpdateAllocStats@vmkernel#nover+0x4b stack:
VmMem_NodeStatsSub@vmkernel#nover+0x59 stack:
VmMem_FreePageNoBackmap@vmkernel#nover+0x8b stack:
VMotionRecv_ReadCompletePages@esx#nover+0x18f stack:

• A virtual machine with recently hot-added memory has been vMotioned to the ESXi host which experienced the PSOD
• Sample PSOD screenshot:

 

 

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 7.0

Cause

This issue is caused by a race condition where a page lookup takes place with the older numa memory layout while the memory hot plug module is recomputing the numa memory layout of a VM on the destination during a vMotion.

 

Resolution

This issue is resolved in ESXi 8.0 U1c patch release, please see VMware ESXi 8.0 Update 1c Release Notes for patching details.
It is also resolved in ESXi 7.0 U3o patch release, please see VMware ESXi 7.0 Update 3o Release Notes for patching details.

Workaround:

The issue can be avoided by disabling the memory hotplug feature of the VMs.
For more information, see the Change Memory Hot Add Settings in the vSphere Virtual Machine Administration guide.