Error: "kernel: BUG: soft lockup - CPU#Y stuck for Xs" within VM

Products

• VMware vSphere ESXi

Issue/Introduction

On VM at the terminal or in the /var/log/message file, the following message appears:

kernel: BUG: soft lockup - CPU#Y stuck for Xs!

where Y is one of the CPU cores and X is an amount of time.  

Cause

Possible causes but not limited to the following include:

• The VM is undergoing a snapshot with quiesce.
• The VM is on an "over committed" host with insufficient RAM, CPU, or disk throughput to support the guests.
• Storage used by the VM is experiencing delays or timeouts attempting to read or write requested data
• Other resource intensive activity on the host or VM that exceeds available capacity

 A soft lockup occurs when a virtual machine's vCPU is unable to run a new task for more than 20 seconds. 

Soft lockups can cause VMs to become unresponsive for short periods of time and trigger application timeouts or failover. VMs that are experiencing a soft lockup might also have unusually high or unusually low CPU utilization, depending on the exact cause of the soft lockup.

Resolution

Possible solutions include the following:

• Investigate the ESXi host and review CPU contention possibilities.  If a host is overcommited, the VM may struggle to have access to the resources it needs.
• Investigate the underlying storage used by affected VM's.
  A timeout while attempting to read or write a requested file will cause a VM to freeze, particularly if a system file or executable file is involved. This occurs whether the storage is handled by ESXi or mounted directly within the guest OS.
  If affected VM's are using datastores, and if those datastores run out of disk space, then the VM's will not be able to perform disk writes anymore. Clear space on datastores, as needed. If vCenter is unavailable, you can increase the datastore size by following the steps from article "Increase or expand the datastore directly from the ESXi host".
• In some cases the soft lockup timer value can be increased within the guest OS.  This increases the time before a soft lockup is triggered, effectively disabling its sensitivity; however, this is generally not recommended as it can mask underlying issues and should only be used for troubleshooting purposes in a controlled environment.
  
  • Adjust the soft lockup timeout value:
    • Access the file: Login to Guest OS as root via SSH/Terminal and navigate to /proc/sys/kernel/softlockup_thresh.
    • Modify the value: (this may vary by linux distribution, consult the linux distributions documention before making any changes)
      • echo new_timeout_value > /proc/sys/kernel/softlockup_thresh
        Note: Replace "new_timeout_value" with the desired time in seconds before a soft lockup is triggered. 
  • Note: 
    • If you don't see softlockup_thresh in /proc/sys/kernel/
      • • Check for watchdog_thresh
        • ls /proc/sys/kernel/watchdog_thresh
    • If it exists: You can change it the same way:
      • echo new_timeout_value >/proc/sys/kernel/watchdog_thresh

Additional Information

Alternative approaches to managing soft lockups:

• Analyze system logs and performance metrics to identify the root cause of the soft lockup, such as a resource bottlenecks. 
• Adjust application settings or resource allocation to minimize potential CPU intensive operations. 
• Update to the latest kernel version as it may include bug fixes related to soft lockup detection.