Windows or Linux virtual machines running on AMD Zen series CPUs crashing due to a doublefault
Article ID: 396667
Updated On:
Products
Issue/Introduction
- Windows or Linux virtual machines running on AMD Zen series CPUs may panic and shut down due to a doublefault exception.
- On Linux systems, the output of dmesg in the virtual machine will contain a backtrace similar to the following:
[XXXXXX.XXXXXX] PANIC: double fault, error_code: 0x0
[XXXXXX.XXXXXX] Call Trace:
[XXXXXX.XXXXXX] <#DF>
[XXXXXX.XXXXXX] ? df_debug+0x1d/0x36
[XXXXXX.XXXXXX] ? do_double_fault+0xe5/0x180
[XXXXXX.XXXXXX] ? double_fault+0x1e/0x30
[XXXXXX.XXXXXX] ? acpi_processor_thermal_init.cold.6+0x66/0x66
[XXXXXX.XXXXXX] ? native_safe_halt+0xe/0x20
[XXXXXX.XXXXXX] </#DF>
[XXXXXX.XXXXXX] acpi_idle_do_entry+0x93/0xa0
[XXXXXX.XXXXXX] acpi_idle_enter+0x5f/0xd0
[XXXXXX.XXXXXX] cpuidle_enter_state+0x86/0x470
[XXXXXX.XXXXXX] cpuidle_enter+0x2c/0x40
[XXXXXX.XXXXXX] do_idle+0x26f/0x2d0
[XXXXXX.XXXXXX] cpu_startup_entry+0x6f/0x80
[XXXXXX.XXXXXX] start_secondary+0x187/0x1d0
[XXXXXX.XXXXXX] secondary_startup_64_no_verify+0xd1/0xdb
Another Example:
[XXXXXX.XXXXXX] traps: PANIC: double fault, error_code: 0x0
[XXXXXX.XXXXXX] double fault: 0000 [#1] PREEMPT SMP NOPTI
[XXXXXX.XXXXXX] CPU: XX PID: 0 Comm: swapper/12 Kdump: loaded Not tainted 6.4.0-150600.23.38-default #1
[XXXXXX.XXXXXX] Hardware name: VMware, Inc. VMware7/440BX Desktop Reference Platform, BIOS VMW7
[XXXXXX.XXXXXX] RIP: 0010:error_entry+0x1a/0x150
...
[XXXXXX.XXXXXX] Call Trace:
[XXXXXX.XXXXXX] <#DF>
[XXXXXX.XXXXXX] ? __die_body+0x1a/0x60
[XXXXXX.XXXXXX] ? die+0x38/0x60
[XXXXXX.XXXXXX] ? exc_double_fault+0x175/0x190
[XXXXXX.XXXXXX] ? asm_exc_double_fault+0x1f/0x30
[XXXXXX.XXXXXX] ? early_xen_iret_patch+0xc/0xc
[XXXXXX.XXXXXX] ? asm_exc_page_fault+0x9/0x30
[XXXXXX.XXXXXX] ? error_entry+0x1a/0x150
[XXXXXX.XXXXXX] </#DF>
- To determine if this is a match, check in the dmesg log with the following command:
grep "double fault" /var/crash/*dmesg* -A20- On Windows systems, the stack trace for the dump file will be similar to the following:
nt!KeBugCheckExnt!KiBugCheckDispatch+0x69nt!KiDoubleFaultAbort+0x2bdhal!HalProcessorIdle+0xfnt!PpmIdleDefaultExecute+0x1bnt!PpmIdleExecuteTransition+0x6bcnt!PoIdle+0x33fnt!KiIdleLoop+0x2c
Environment
Windows or Linux virtual machines running on ESXi 7.0 U2 or later releases and AMD Zen series CPUs
Cause
This issue is caused by a race condition in the hypervisor on AMD hosts during the NUMA migration of a VM's internal data. It is very rare.
Resolution
ESX 9.1 and later releases are not impacted by this issue. A fix for ESX 8.0 U3 is being planned. This article will be updated when the patch is available.
Workaround (until the fix is installed):
The problem only occurs when the hypervisor automatically migrates a VM's internal data between NUMA nodes. You can avoid it by turning that feature off on each affected host. No reboot or VM restart is needed.
Using esxcli:
esxcli system settings advanced set -o /Numa/MonMigEnable -i 0
To check the setting (Int Value should be 0):
esxcli system settings advanced list -o /Numa/MonMigEnable
Notes:
-Guest memory placement is not affected. Only the hypervisor's own data for a VM stays on its original NUMA node.
-Most workloads see no impact. Workloads with many TLB misses (large, randomly accessed memory) may see a performance impact of at most 5%.
-To return to the default, set the value to 1.
-Apply the setting to every AMD host that runs the affected release. After upgrading to the fixed release, it is no longer needed.
Feedback
