Following storage outage, Linux VM CPU lockup and VSCSI resets when fstrim runs
Article ID: 435667
Updated On:
Products
Issue/Introduction
- Following storage outage VMs experience filesystem corruption requiring manual fsck or VM restart.
- During next guest OS fstrim (UNMAP) operation the VM hangs with CPU errors:
watchdog: BUG: soft lockup - CPU## stuck for xxs!
Environment
VMware vSphere ESX 8.0
Cause
During the storage outage, the UNMAP command failed to complete its execution on the backing LUN, resulting in stale metadata locks.
Because the UNMAP process requires exclusive locks to safely perform block reclamation, these orphaned locks are now preventing subsequent space reclamation tasks and causing transaction errors.
/var/run/log/vmkernel.log reports
vmkernel: cpu##:#######)VSCSI: 3772: handle ###################'(GID:####)(vscsi0:0):processing reset for handle ... state ##########
vmkernel: cpu##:#######)Res6: 2944: '<datastorename>': RC Lock not free for type 1, return TXN FULL
vmkernel: cpu##:#######)Fil6: 3816: <datastorename>: <FD c## r##> - Failed to unmap file blocks 0/1:Transaction ran out of lock space or log space
Resolution
- If the guest OS hangs, a hard reset is required to clear the vSCSI reset loop.
- Migrate the affected VMs to a different datastore as moving the VM disk files forces the storage layer to release/clear the stale metadata locks associated with the specific blocks on the original datastore.
- Perform a staged, rolling reboot of all ESXi hosts in the cluster.
Additional Information
Feedback
