vSAN VM experienced latency issues when moved to specific ESXi Host

Products

• VMware vSAN

Issue/Introduction

Symptoms: 

• vSAN VM is having a high IO latency issue.

Environment

vSAN 7.X
vSAN 8.X

Cause

 There is a high TCP retransmission rate on the vSAN vmkernel and also, and there are TCP/Errors for the same vmkernel adapter.

We see there is a constant increase in the packet drops for vSAN network uplinks:

Example : 

esxcli network nic stats get -n vmnic3

   NIC statistics for vmnic3:
      Packets received: 23751368617
      Packets sent: 19730325195
      Bytes received: 130908835599844
      Bytes sent: 68318281765261
      Receive packets dropped: 56035123
      Transmit packets dropped: 0
      Multicast packets received: 261269
      Broadcast packets received: 259237
      Multicast packets sent: 0
      Broadcast packets sent: 1506
      Total receive errors: 0
      Receive length errors: 0
      Receive over errors: 0
      Receive CRC errors: 0
      Receive frame errors: 0
      Receive FIFO errors: 0
      Receive missed errors: 0
      Total transmit errors: 0
      Transmit aborted errors: 0
      Transmit carrier errors: 0
      Transmit FIFO errors: 0
      Transmit heartbeat errors: 0
      Transmit window errors: 0

Resolution

Please refer to the fix mentioned in KB # High vSAN latency Observed for vSAN VMs