Is there any impact on VM performance when adding new vSAN Disk to existing disk groups?

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Creating new vSAN disk groups or adding nodes to a vSAN cluster may result in high resynchronization activity and there is a need to proactively control the vSAN resync operation to ensure that a controlled amount of bandwidth is allocated for vSAN resync activity.
  
  
• Proactively throttle vSAN Resync on vSAN 7.X vSAN nodes to ensure there is no impact on VM performance.

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.x

 

Resolution

vSAN should automatically take care of the vSAN disk imbalance in the cluster and will start resyncing vsan objects. In case if you want to proactively prevent VM performance issues. Follow the below steps. 

To check the current resync throughput at the host level, connect to every ESXi host in the vSAN cluster via putty session and run the below command. 

esxcli vsan resync throttle get 

The default output will be Level: 0, which means that currently there is no throttling set on the vSAN node. 

To change the vSAN resync throttle on vSAN Nodes, for example To throttle resync traffic to not exceed 10 Mbps on the ESXi host then we need to run the following command on all the ESXi hosts.

Note: Always make sure the resync throttling is set on all the ESXi host to ensure we have uniform vSAN resync operation across the vSAN nodes. 

esxcli vsan resync throttle set --level <0-512mb> 

Example: 

esxcli vsan resync throttle get;
esxcli vsan resync throttle set --level=10  
esxcli vsan resync throttle get;

To set the vSAN resync throttle back to default values 

esxcli vsan resync throttle get;
esxcli vsan resync throttle set --level=0;
esxcli vsan resync throttle get;

Note: Resync operation on the vSAN Cluster will take a longer time since the vSAN resync bandwidth is set to a lower value at the same time.

vSAN Resync throttle should not be set to higher bandwidth then this can lead to vSAN congestion hence it is always recommended to start with a lower value and increment the vSAN resync throttle value to a smaller incremental value and monitor the vSAN cluster behavior to ensure we don't see any abnormal vSAN congestion or VM facing performance issue. 

Example: For the 1st time set the vSAN resync throttle value to 5 Mbps allow this to run for some time and check the resync speed by running the command "esxcli vsan debug resync summary get" and then check if the current vSAN resync rate is ok. If the VM operation is not impacted and you need more resync speed then you can set "esxcli vsan resync throttle set --level=15" on all the vSAN nodes and check if this ok. 

Example : 

esxcli vsan debug resync summary get  
   Total Number Of Resyncing Objects: 0
   Total Bytes Left To Resync: 0
   Total GB Left To Resync: 0.00