Symptoms:
vSAN strongly depends on the underlying Network Performance to produce acceptable overall performance values.
For troubleshooting purposes one may have to measure the Network Performance of the VSAN Network.
The steps in Resolution below will explain how to do it, with the built-in tool available on ESXi.
[root@esx03-nla1:/usr/lib/vmware/vsan/bin] ./iperf.copy -s -B 193.xx.xx.xx -i 1
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 193.xx.xx.xx
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 193.xx.xx.xx port 5001 connected with 193.xx.xx.xx port 12677
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 866 MBytes 7.27 Gbits/sec
[ 4] 1.0- 2.0 sec 937 MBytes 7.86 Gbits/sec
[ 4] 2.0- 3.0 sec 921 MBytes 7.72 Gbits/sec
[ 4] 3.0- 4.0 sec 930 MBytes 7.80 Gbits/sec
[ 4] 4.0- 5.0 sec 940 MBytes 7.88 Gbits/sec
[ 4] 5.0- 6.0 sec 933 MBytes 7.83 Gbits/sec
[ 4] 6.0- 7.0 sec 952 MBytes 7.98 Gbits/sec
[ 4] 7.0- 8.0 sec 936 MBytes 7.85 Gbits/sec
[ 4] 8.0- 9.0 sec 936 MBytes 7.85 Gbits/sec
[ 4] 9.0-10.0 sec 879 MBytes 7.38 Gbits/sec
[ 4] 0.0-10.0 sec 9.01 GBytes 7.74 Gbits/sec
[root@esx04-nla1:/usr/lib/vmware/vsan/bin] ./iperf.copy -c 193.xx.xx.xx -fM
------------------------------------------------------------
Client connecting to 193.xx.xx.xx, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 193.xx.xx.xx port 12677 connected with 193.xx.xx.xx port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9230 MBytes 923 MBytes/sec
[root@esx04-nla1:/usr/lib/vmware/vsan/bin]
esxcli network firewall set --enabled true