VMs failed to load OS after reboot due to vSAN LSOM SSD Congestion
Article ID: 443857
Updated On:
Products
Issue/Introduction
- Virtual Machines fail to boot after a restart and may stuck in OS loading state.
- vobd.log indicate LSOM SSD congestion and show exceeded the predefined threshold (200):
[vSANCorrelator] [vob.vsan.lsom.congestionthreshold] LSOM SSDCong in [UUID] Congestion State: Exceeded. Congestion Threshold: 200 Current Congestion: 204.
Environment
VMware vSAN 7.0.x / 8.0.x (Original Storage Architecture).
Deduplication and Compression enabled.
Cause
One or more capacity drives exhibiting intermittent high I/O latency of up to seconds. This slow-drain behavior caused in the vSAN cache tier, leading to the high LSOM congestion state.
- vmkwarning.log show extreme latency spikes for specific capacity devices within a disk group:
WARNING: ScsiDeviceIO: 1515: Device [Device_ID] performance has deteriorated. I/O latency increased from average value of 523 microseconds to 3062123 microseconds.
Resolution
Workaround:
To restore VM operations, the congestion must be mitigated by removing the high-latency component from the data path.
Identify the Impacted Disk Group
Review the vobd.log to find the Disk Group UUID associated with the SSDCong exceeded message.
Cross-reference the vmkwarning.log to confirm which physical device is reporting high latency.
Unmount the Affected Disk Group using the command
'esxcli vsan storage diskgroup unmount -s [Cache_Disk_NAA_ID]'. Refer Managing and Configuring a vSAN disk group using esxcli commands for more detailed instructions.
By unmounting the disk group, it removes the bottleneck. vSAN will then satisfy I/O requests using remaining healthy replicas on other hosts.
Hardware Replacement
Coordinate with hardware vendor to replace the faulty capacity drive identified in the logs. Once the hardware is replaced, the disk group can be recreated.
Feedback
