vSphere Fault Tolerance protected Linux VMs on HWv20 and later with the VMCI DMA datagrams feature may lockup or panic on ESXi8.0/8.0U1/8.0U2

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

On ESXi 8.0/8.0U1/8.0U2, if Linux Virtual Machine:

• Is protected by vSphere Fault Tolerance(FT).
• And Virtual Machine is on HWv20 or later and the Linux kernel has VMCI DMA datagrams feature.

The Virtual Machine may suffer soft lockups, or kernel panic(which happens on some supported distros if kernel.softlockup_panic is set as 1).

Guest OS console shows the below messages

[  948.008390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
...
[  976.011365] watchdog: BUG: soft lockup - CPU#0 stuck for 21s!

or 

[    6.160741] piix4_smbus 0000:00:07.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
 rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
 rcu: o1-...0: (1 GPs behind) idle=392/1/0x4000000000000000 softirq=7926/7937 fqs=14965
 rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
 rcu: o1-...0: (1 GPs behind) idle=392/1/0x4000000000000000 softirq=7926/7937 fqs=59751
 INFO: task khugepaged:35 blocked for more than 120 seconds.
       Not tainted 4.18.0-477.10.1.el8_8.x86_64 #1

   

 To determine the virtual hardware version of Virtual Machine, select the Virtual Machine from the vSphere Client and click the Summary tab. The Compatibility field displays the virtual hardware version. For instance, "ESXi 8.0 and later (Virtual Machine version 20)" in the filed means the Virtual Machine is on HWv20.

Note:

• Non-FT Virtual Machines are not impacted.
• FT-Virtual Machines older than HWv20, or without VMCI DMA datagrams feature is also not impacted.

The VMCI DMA datagrams feature was supported starting from upstream Linux 5.18. Linux distros supporting this feature includes:

RHEL:8.7/9.1 and later

Ubuntu: 22.04.1 LTS (kernel 5.15.0-41.44 onwards onwards) and 22.10 LTS

SUSE SLES: SLES15 SP3 (kernel 5.3.18-150300.59.93.1 onwards) and SLES15 SP4

Photon: Photon 4.0

 

Environment

VMware vSphere ESXi 8.0
VMware vSphere ESXi 8.0.1
VMware vSphere ESXi 8.0.2

Cause

The VMCI DMA datagrams feature is incompatible with Fault Tolerance. With Fault Tolerance turned on, the Linux Kernel vmw_vmci driver for the VMCI device ends up in a busy loop, causing the Virtual Machine to enter a soft lockup if the VMCI device is used.

Resolution

This issue is fixed in release ESXi 8.0 U3 build 24022510.

Workaround for ESXi 8.0, 8.0.1,8.0.2:

Post verifying FT Virtual Machine is impacted by checking virtual hardware version and GuestOS version, please follow the below mentioned steps to reconfigure the FT Virtual Machine:

1. Power-Off the FT Virtual Machine

2. Add the attribute/value vmci.dmaDatagramSupport = FALSE to Virtual Machine through vSphere Client. Please refer Set Advanced Virtual Machine Attributes for information on setting the advanced attributes for Virtual Machine .

3. Power-On the FT Virtual Machine.