SSD congestion due to failing storage disks/controller.
Article ID: 408799
Updated On:
Products
Issue/Introduction
Symptom:
- VMs may experience intermittent or permanent freezing, leading to unresponsiveness.
- These performance issues typically manifest immediately following a lightning-induced power outage or sudden power disruption.
- SSD congestion is observed in a vSAN Cluster and all disks residing on the same storage controller show read/write failures.
SSD congestion was collected via the following one liner.
while true; do echo "================================================"; date; for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do echo $ssd;vsish -e get /vmkModules/lsom/disks/$ssd/info|grep Congestion;done; for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do llogTotal=$(vsish -e get /vmkModules/lsom/disks/$ssd/info|grep "Log space consumed by LLOG"|awk -F \: '{print $2}');plogTotal=$(vsish -e get /vmkModules/lsom/disks/$ssd/info|grep "Log space consumed by PLOG"|awk -F \: '{print $2}');llogGib=$(echo $llogTotal |awk '{print $1 / 1073741824}');plogGib=$(echo $plogTotal |awk '{print $1 / 1073741824}');allGibTotal=$(expr $llogTotal \+ $plogTotal|awk '{print $1 / 1073741824}');echo $ssd;echo " LLOG consumption: $llogGib";echo " PLOG consumption: $plogGib";echo " Total log consumption: $allGibTotal";done;sleep 30; done
Example
storage core device stats get
naa.##########549700 Device: naa.##########549700 Successful Commands: 7541554 Blocks Read: 35393757 Blocks Written: 42388956 Read Operations: 3797796 Write Operations: 3741974 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 48887 Failed Blocks Read: 58248736 Failed Blocks Written: 71827488 Failed Read Operations: 6646 Failed Write Operations: 42234 Failed Reserve Operations: 0
naa.##########5496e0 Device: naa.##########5496e0 Successful Commands: 7511603 Blocks Read: 35130001 Blocks Written: 42371440 Read Operations: 3768446 Write Operations: 3741371 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 48698 Failed Blocks Read: 56475648 Failed Blocks Written: 73730080 Failed Read Operations: 6307 Failed Write Operations: 42385 Failed Reserve Operations: 0
naa.##########549710 Device: naa.##########549710 Successful Commands: 7557820 Blocks Read: 35155268 Blocks Written: 42740640 Read Operations: 3768906 Write Operations: 3787124 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 49338 Failed Blocks Read: 58300928 Failed Blocks Written: 71985248 Failed Read Operations: 6812 Failed Write Operations: 42518 Failed Reserve Operations: 0
naa.##########5496f0 Device: naa.##########5496f0 Successful Commands: 14001031 Blocks Read: 450457838 Blocks Written: 41458280 Read Operations: 10129220 Write Operations: 3870029 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 89696 Failed Blocks Read: 1296358928 Failed Blocks Written: 57525248 Failed Read Operations: 46238 Failed Write Operations: 43445 Failed Reserve Operations: 0
naa.#########53c789 Device: naa.##########53c789 Successful Commands: 23053319 Blocks Read: 116976309 Blocks Written: 343105728 Read Operations: 5079311 Write Operations: 17972076 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 247689 Failed Blocks Read: 220614144 Failed Blocks Written: 667578068 Failed Read Operations: 29980 Failed Write Operations: 217645 Failed Reserve Operations: 0
naa.##########549770 Device: naa.##########549770 Successful Commands: 13566442 Blocks Read: 424205607 Blocks Written: 39037068 Read Operations: 9999472 Write Operations: 3565194 Reserve Operations: 0 Reservation Conflicts: 0 Failed Commands: 86038 Failed Blocks Read: 1185489944 Failed Blocks Written: 55110144 Failed Read Operations: 43902 Failed Write Operations: 42130 Failed Reserve Operations: 0
Environment
VMware vSAN( All Versions)
Cause
Due to read and write failures on all disks presented to ESXi via the storage controller that is failing/failed. vSAN was not able to de-stage data to the impacted capacity drives correctly leading to write buffers of the Cache disks filling and SSD congestion was introduced to relieve the I/O bottleneck.
Congestion is a flow control mechanism used by vSAN. Whenever there is a bottleneck in a lower layer of vSAN (closer to the physical storage devices), vSAN uses this flow control (aka congestion) mechanism to relieve the bottleneck in the lower layer and instead reduce the rate of incoming I/O at the vSAN ingress, (i.e. vSAN Client VMs). This reduction of the incoming rate is done by introducing an I/O delay at the ingress that is equivalent to the delay the I/O would have occurred due to the bottleneck at the lower layer. Thus, it is an effective way to shift latency from the lower layers to the ingress without changing the overall throughput of the system. vSAN measures congestion as a scalar value between 0 to 255, and the introduced delay is computed using a randomized exponential backoff method, based on the congestion metric.
Resolution
Place impacted host in to Maintenance Mode via ensure accessibility and reach out to your hardware vendor to investigate the failed/failing disks and storage controller.
If you have any additional questions, or concerns regarding this issue please open a case with vSAN Support for further investigation.
Additional Information
Reference kb
Feedback
