RHEL guest kernel panic during vCPU hot-add on AMD EPYC hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Red Hat Enterprise Linux (RHEL) virtual machines may encounter a kernel panic when increasing the vCPU count on hosts equipped with AMD EPYC processors. This issue stems from a software race condition within the guest operating system's scheduler and is not a hypervisor or hardware defect.

• Virtual machine terminates or reboots immediately following a vCPU hot-add operation.
• Guest OS crash dumps (vmcore) show a NULL pointer dereference.
• The call trace contains the __update_blocked_fair function.
• Log snippet from the guest: BUG: kernel NULL pointer dereference, address: 0000000000000000 RIP: 0010:__update_blocked_fair+0xd1/0x4b0 Call Trace: update_blocked_averages+0x238/0x5c0 run_rebalance_domains+0x7d/0xe0
• The ESXi vmware.log reports: The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

 

Environment

• Hypervisor:  8.x
• Hardware: AMD EPYC processors (Zen 2, Zen 3, Zen 4)
• Guest OS: Red Hat Enterprise Linux 9.x (specifically version 9.7)

Cause

A race condition exists in the RHEL kernel scheduler during the update of blocked load tracking when new vCPUs are added to the system. This is a guest operating system defect documented in Red Hat Solution 7082530.

Resolution

1. Analyze the guest OS vmcore to confirm the __update_blocked_fair NULL pointer dereference.
2. Correlate the timing of the crash with any vCPU hot-add tasks in vCenter.
3. Contact Red Hat Support to obtain and apply the kernel patch or update associated with Red Hat Solution 7082530.
4. If hot-add is required before a patch is applied, perform vCPU increases while the virtual machine is powered off.