3rd party Hyper Converged Infrastructure controller virtual machines (CVM) running on ESXi 6.7, 7.0.x and 8.0.x may experience soft lockups and experience unexpected reboots

Products

• VMware vSphere ESXi

Issue/Introduction

• 3rd party Hyper Converged Infrastructure controller virtual machines (CVM) running on ESXi 6.7/7.0 and 8.0 may experience soft lockups and experience unexpected reboots.
• The OS log files will have messages similar to the below when these reboots occur.        "NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s!"
• "vmware.log" for 3rd party Hyper Converged Infrastructure  controller VM's, will have messages similar to below:

 YYYY-MM-DDThh:mm:ss.232Z In(05) vcpu-0 - Vix: [vmxCommands.c:7182]:                 VMAutomation_HandleCLIHLTEvent. Do nothing.
 YYYY-MM-DDThh:mm:ss.232Z In(05) vcpu-0 - MsgHint: msg.monitorevent.halt
 YYYY-MM-DDThh:mm:ss.232Z In(05)+ vcpu-0 - The CPU has been disabled by the guest      operating system. Power off or reset the virtual machine.
 YYYY-MM-DDThh:mm:ss.232Z In(05)+ vcpu-0 - ---------------------------------------
 YYYY-MM-DDThh:mm:ss.235Z In(05) vcpu-0 - VigorTransportProcessClientPayload:     opID=vim-cmd-6d-8fa4 seq=49288: Receiving Bootstrap.MessageReply request.

NOTE: This KB is not be used for any other VM soft lockup issue.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

There is a circular dependency between a resource provided by a VM and the ESXi host consuming this resource (NFS mount).

Resolution

This issue is mitigated in VMware vSphere ESXi 7.0 U3n and ESXi 8.0 U1c patches.

To download go to the Broadcom Portal https://support.broadcom.com/web/ecx page sign in, go to my Downloads,  VMware vSphere choose your Product, then the version.
 

Workaround:

To workaround use one of the below steps  to disable VMX logging for 3rd party Hyper Converged Infrastructure controller VM's.

1. Disabling virtual machine logging (For 3rd party Hyper Converged Infrastructure controller VM's) using SSH. 

• Power off the CVM virtual machine.
• Navigate to  VM folder using command cd /vmfs/volumes/datastore/ServiceVM_Centos/

           Edit the VMX file using VI editor.  Add the following line in the .vmx file.
            logging = FALSE

•  Power on the VM.

2. From GUI - To configure the disable logging option:

• Login to VMware vSphere Client or ESXi host client and power off the virtual machine.
• Right click the virtual machine and click Edit Settings.
• Select the VM Options tab.
• Click Advanced > General.
• Under Settings, Uncheck the Enable logging option. (De-select the Enable logging option to disable.)
• Power on the virtual machine.

Example:
 

Additional Information

Impact/Risks:
The workaround will disable logging to vmware.log .But it has no noticeable impact on the VM behavior, especially no negative performance impact.