Poor vSAN performance when copying the data from physical server to the virtual server running on vSAN

Products

• VMware vSAN

Issue/Introduction

Symptoms:

When copying the data between physical server to virtual server, the copy operations may degrade significantly.

The data transfer rate would drop gradually as the copy task progresses. 

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

The issue is seen when the underlying vSAN disks are experiencing physical disk latency. This physical disk latency is causing the vSAN disk group SSD congestion as the operations is continuous writes on to vSAN disks.

This can be monitored using esxtop command and looking for the disks DAVG values by pressing 'u' on esxtop screen.

Also, upon checking for the vSAN health, you may see SSD Congestion warnings start showing up.

You can also login into SSH session for each ESXI host of vSAN cluster and run the command below to check the congestion level.

for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do echo $ssd;vsish -e get /vmkModules/lsom/disks/$ssd/info|grep Congestion;done

Example output as below.

524938eb-705e-164f-b5b2-####
   memCongestion:0
   slabCongestion:0
   ssdCongestion:163
   iopsCongestion:0
   logCongestion:0
   compCongestion:0
   mdCongestion:0 

Resolution

In order to resolve the issue or improve the performance for the data transfer operation, we should fix the underlying hardware issues. If the HDD or SSD experiencing latency, involve hardware vendor to understand the cause of the disks latency.