High read/write latency is observed on multiple hosts in a vSAN cluster

Products

• VMware vSAN

Issue/Introduction

• High read/write latency is observed on multiple hosts in a vSAN cluster
• Placing those hosts into maintenance mode results in the read/write latency migrating to other hosts in the cluster indicating the latency is cluster wide and not specific to a host
• Looking at the vSAN Perf Graphs via vCenter > Cluster > Monitor > vSAN > Performance > Backend no bottleneck or latency is observed on any of the disks or the network for any of the hosts in the cluster
• When looking at historical performance data via Aria Operations it's observed there is a pattern to the read/write latency where it's occurring at a specific time daily or weekly and then returning to normal. 
  

Environment

VMware vSAN (All Versions)

Cause

When read/write latency is occurring across the entire cluster at specific times daily or weekly and then returning to normal with no vSAN Backend latency observed this is an indicator that an application or task in the environment is causing an increase in workload such as backups, antivirus, ect..

Resolution

Work with your application team to determine what the application/task is that's causing the increase in workload at these specific times.

If it is an antivirus or backup application/task try staggering the tasks to ease the workload by not doing backups/scans of all or too many VMs at once.