High write latency is observed on multiple VMs in a vSAN cluster

Products

• VMware vSAN

Issue/Introduction

• High read/write latency is observed on multiple VMs in a vSAN cluster
• Looking at the vSAN Perf Graphs via vCenter > Cluster > Monitor > vSAN > Performance > Backend & VM the latency observed spikes & settles simultaneously with each other.
  
  
  
  
  
  

Environment

VMware vSAN (All Versions)

Cause

When write latency, congestion, & outstanding I/O is occurring across multiple VMs and the vSAN Backend latency, congestion, & outstanding I/O coincides with the VM latency and then the environment returns to normal functionality this is an indicator that there is an increase in workload resulting in a large amount of outstanding IO vSAN needs to process.

This can be caused by a scripted job running in the environment introducing a spike in throughput and IOPs, such as deleting a large amount of VM snapshots simultaneously.

Resolution

Decrease the amount of tasks the script is processing in the environment.