Windows VMs are crashing into BSOD 0x50 when running on a specific ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines (VMs) with a Windows guest operating system are crashing with a Bluescreen (BSOD) when running on a specific ESXi host.
• The BSOD displays an exception 0x50.
• The VMX log, /vmfs/volumes/<datastore>/<vm_name>/vmware-##.log, has entries similar to:

  <time_stamp> Wa(03) vcpu-7 - WinBSOD: Synthetic MSR[0x40000100] 0x50

• The same VM runs stable when located on other ESXi and does not experience any BSOD.

Environment

• VMware vSphere ESXi 7.0.x
• VMware vSphere ESXi 8.0.x
• VMware ESX 9.x

Cause

The exception code 0x50 stands for PAGE_FAULT_IN_NONPAGED_AREA. This exception can be caused by a number of different problems, either within the guest OS, such as:

• Installation of a faulty system service in the guest operating system
• Faulty driver code in the guest OS
• Antivirus software using a faulty filter driver
• Corruption in the NTFS file system on any of the volumes mounted in the OS

Or by a fault in the physical CPU, memory controller or one of the physical memory modules, such as a bit flip. A bit flip occurs when a single bit in the memory unexpectedly changes its state from 1 to 0 or from 0 to 1 and is unsually caused by hardware fault.

Resolution

This is not a VMware issue, but either a fault in the guest operating system or within the underlying physical hardware platform.
To isolate which of these is at fault, migrate the affected VM to a different ESXi host and monitor if the same exception occurs there:

1. If the issue repeats while the VM is running on the new host, the issue is caused by a component within the guest operation system. Involve the guest OS vendor for further troubleshooting, with emphasis on drivers, filter drivers (e.g. from an Antivirus application) and potential file system issues.
2. If the issue does not occur while the virtual machine is running on the new host, guest OS internal causes can be ruled out. Involve the server vendor for a thorough hardware check, specifially for the physical CPUs, memory controller and the memory modules.

Additional Information

For more information related to this specific Windows expection, refer to the Microsoft article Bug Check 0x50: PAGE_FAULT_IN_NONPAGED_AREA.