High Write Latency and Cache Tier Disk Failure in vSAN OSA
search cancel

High Write Latency and Cache Tier Disk Failure in vSAN OSA

book

Article ID: 415297

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

High write latency and performance degradation in a vSAN OSA cluster are often caused by physical hardware failures in the cache or capacity tiers. This article provides steps to diagnose and resolve disk-related performance issues.

Symptoms

  • From vSAN cluster > Monitor > vSAN > Performance > Write latency observed.

  • VMs on the affected vSAN datastore are experiencing slow performance. 

  • Extreme degraded performance can lead to VMs going hung or inaccessible.

  • The vSAN cluster experiences intermittent or persistent network latency, requiring an investigation of the network uplinks

  • Powering on the vCenter VM results in error: Failed to create temporary file—file already exists / failed to start VM (error -18).

  • High write latency is observed on specific hosts within the vSAN cluster.

  • esxtop on affected hosts shows high DQLEN or latency for vSAN disks.

Evidence of this failure is found in /var/run/log/vsandevicemonitord.log:

From ESXi, var/run/log/vsandevicemonitord.log reports disk with "DISK_UNHEALTHY_LATENCY" events:

2025-10-13T09:41:24Z In(14) vsandevicemonitord[2099770]: [779641398784]: WARNING - WRITE Average Latency on VSAN device naa.############### has exceeded threshold value 2500000 us 1 times.
2025-10-13T09:41:24Z In(14) vsandevicemonitord[2099770]: [779641398784]: Latency monitoring interval # is 0 for device
2025-10-13T10:11:30Z In(14) vsandevicemonitord[2099770]: [779641398784]: Device naa.###############  state is DISK_UNHEALTHY_LATENCY SMART.

2026-01-21T02:18:57Z In(14) vsandevicemonitord[2099454]: [604934988480]: WARNING - WRITE Average Latency on VSAN device naa.############## has exceeded threshold value 200000 us 1 times.
2026-01-21T02:18:57Z In(14) vsandevicemonitord[2099454]: [604934988480]: Latency monitoring interval # is 3 for device naa.###############
2026-01-21T02:28:59Z In(14) vsandevicemonitord[2099454]: [604934988480]: WARNING - WRITE Average Latency on VSAN device naa.############### has exceeded threshold value 200000 us 1 times.

Under /var/run/log/vmkernel.log, there will be 'performance is deteriorated' messages for these devices:

2026-08-02T05:13:48.369Z cpu7:2097941)WARNING: ScsiDeviceIO: 1513: Device naa.############### performance has deteriorated. I/O latency increased from average value of 333 microseconds to 191507 microseconds.
2026-08-02T05:14:04.981Z cpu34:2097937)WARNING: ScsiDeviceIO: 1513: Device naa.############### performance has deteriorated. I/O latency increased from average value of 333 microseconds to 389592 microseconds.

Environment

VMware vSAN 7.x
VMware vSAN 8.x

Cause

Physical media degradation, unrecoverable medium errors, or controller-level I/O timeouts on vSAN cache or capacity tier disks. These issues may also occur due to unsupported hardware configuration not listed in the VMware Compatibility Guide.  High write latency on a vSAN cache tier disk prevents the host from completing fundamental I/O operations. This degradation is frequently caused by a failing hardware component or the use of unsupported hardware not listed in the VMware Compatibility Guide.

 

Resolution

  1. Verify HCL Compliance: Visit the VMware Compatibility Guide to ensure all SSDs and NVMe devices in the cluster are officially supported for the running ESXi build.
  2. Identify Faulty Disk: Review /var/run/log/vsandevicemonitord.log for the naa ID of the device reporting DISK_UNHEALTHY_LATENCY.
  3. Check Physical Logs: Inspect /var/run/log/vmkernel.log for SCSI-related medium errors or controller timeouts associated with the identified device.
  4. Remove Disk Group:
    • If the faulty disk is a cache tier device, or if Deduplication and Compression are enabled, remove the entire disk group from vSAN.
    • If the disk is a capacity tier device without Deduplication and Compression, remove only the failed disk from the disk group.
  5. Replace Hardware: Dispatch hardware support to replace the physical drive.
  6. Re-add Components: Re-add the disk or disk group to the vSAN cluster and monitor for successful rebalancing.
  7. Power On Virtual Machines: Once the cluster reports a healthy state, power on impacted VMs.

For detailed replacement procedures, refer to:

Replace a Capacity Device in vSAN Cluster
Replace a Flash Caching Device in vSAN Cluster

Additional Information

To speak with a customer representative or a Support Engineer, see Contact Support. 

Powered by Wolken Software