VM using Directpath I/O or SR-IOV experiences a soft lock up and goes unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

• VM using Directpath I/O or SR-IOV experiences a soft lock up and goes unresponsive
• This may recover on its own or would require a reset.

Environment

vSphere ESXi 7.X

vSphere ESXi 8.X

Cause

A rare race condition between the interrupt virtualization and the VMkernel CPU scheduler in  VMs with Directpath I/O device might result in guest kernel soft lockups.

 

Resolution

Resolved in VMware ESXi 8.0 Update 3, available under Download Broadcom products and software
Resolved in VMware ESXi 7.0 Update 3v, available under Download Broadcom products and software

 

Workarounds

• ESXi 7.0.x Workaround
• ESXi 8.0.x Workaround

 

ESXi 7.0.x Workaround

1. Open a SSH session to ESXi  host.

2. Check the current value of the option vtdEnableIntrVirt in the ESXi boot-time options in the VMkernel.boot.* namespace,using the following command:

   # esxcfg-advcfg --get-kernel vtdEnableIntrVirt

    

3. Set a new value for an option using the esxcfg-advcfg command:

   # esxcfg-advcfg --set-kernel "FALSE" vtdEnableIntrVirt

4. Verify that the option was correctly set:

   # esxcfg-advcfg --get-kernel vtdEnableIntrVirt

5. Reboot the ESXi host to apply the changed setting.

6. Once the server is rebooted, run the command  esxcfg-advcfg --get-kernel vtdEnableIntrVirt again, to verify that the value still shows as FALSE.

 

ESXi 8.0.x Workaround

1. Open a SSH session to ESXi  host. 
2. Check the current value of the option iovEnablePostedIntr in the ESXi boot-time options in the VMkernel.boot.* namespace, using the following command:

   # esxcfg-advcfg --get-kernel iovEnablePostedIntr

3. Set a new value for an option using the esxcfg-advcfg command:

   # esxcfg-advcfg --set-kernel "FALSE" iovEnablePostedIntr

4. Verify that the option was correctly set:

   # esxcfg-advcfg --get-kernel iovEnablePostedIntr

5. Reboot the ESXi host to apply the changed setting.
6. Once the server is rebooted, run the command esxcfg-advcfg --get-kernel iovEnablePostedIntr to verify if the value still shows as FALSE


Additional Information

Note: In VMware vSphere ESXi 7.0 Update 2 and later 7.0 version, the default value of vtdEnableIntrVirt is set to TRUE.