VMs running on vSAN become hung and unresponsive with drives reporting high DAVG latency.
Article ID: 450395
Updated On:
Products
Issue/Introduction
- Virtual Machines residing on a vSAN datastore become hung, unresponsive, or experience severe read/write latency.
- vSphere performance charts and esxtop show sustained, massive spikes in Device Average Latency (DAVG) on specific host disk(s).
- Overall disk throughput drops to less than 1 MB/s, despite vSAN health checks and SMART data reporting the physical drives as "Healthy."
- This affects environments using Micron 5x00 series solid-state drives (or OEM variants from vendors like HPE, Cisco, or Lenovo) as vSAN Cache-tier and/or Capacity-tier devices
- vSAN Disk-Groups may report high levels of SSD congestion.
- vSAN Cache-tier and/or Capacity-tier devices may be marked as failed/offline by vSAN DDH (Dying Disk Handling) due to sustained excessively high latency (Dying Disk Handling (DDH) in vSAN).
Environment
VMware vSAN
Cause
A drive firmware issue occurs when the drive's Power-On Hours (POH) counter exceeds 65,535 hours (approximately 7.5 years of uptime).
localcli storage core device smart get -d naa.################
SMART Data for Disk : naa.################Parameter Value Threshold Worst Raw-----------------------------------------------------------Health Status OK N/A N/A N/ARead Error Count 100 50 100 0Power-on Hours 100 0 100 65587 <===
Resolution
This issue is believed to also impact environments using servers from other vendors using Micron 5x00 series drives. The fix requires applying a manufacturer-provided firmware update.
Dell:
PowerEdge: Micron 5x00 Series solid state drive Performance Degradation after 65,535 POH | Dell UK
Feedback
