Ubuntu VM crashes after Kernel Updates to 5.15.0-156-generic

Products

• VMware vSphere ESXi

Issue/Introduction

• VMs crashes abruptly after kernel update to 5.15.0-156-generic.
• No evidence will be seen in the VMware.log of the VM and only we will see tools timeout as below :
  

  XXXXTIMEZ In(05) vcpu-8 - VMXNET3 user: Quiesce device 0.
  XXXXTIMEZ In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
  XXXXTIMEZ In(05) vmx - Tools: [AppStatus] Last heartbeat value 6168945 (last received 14s ago)
  XXXXTIMEZ In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
  XXXXTIMEZ In(05) vcpu-0 - Tools: Tools heartbeat timeout.
  

• There will be no events in hostd/vmkernel logs as well.

Environment

7.x

8.x

Cause

This is a known issue in this version of kernel and it causes CPU soft lockups. The only way to check is from the OS end. 

Resolution

Engage the OS vendor for an appropriate fix or revert the kernel version back to working one.