Isolating VM Performance Problems on HCX L2 Extensions Undetected by HCX Transport Analytics

Products

VMware HCX

Issue/Introduction

You are experiencing performance issues/poor performance with the guest OS/virtual machines using the HCX Layer 2 extension.
When running the HCX perf tests, the results are expected for the HCX NE VMs. For more information on how to run the perftest in HCX, please see Broadcom KB: Steps to Run Perftest in HCX.
- If you are observing performance issues/unexpected results while running the HCX perftest as described in the KB article "Steps to Run Perftest in HCX," the guest OS/virtual machines will be affected, and troubleshooting should be based on the underlay network performance observed during the perftest.
- If the results during the perftests are expected, please use the steps in this KB to isolate if the problem is related to the guest OS configuration.

Environment

VMware HCX

Cause

OS specific parameter changes, such as network adapter performance tuning.

Resolution

Since this is due to OS level parameter changes, engage the OS vendor to identify the cause.

Perform these steps to isolate the problem:

Deploy UPSA (Lightweight Linux VMs with iperf installed) VMs in each Source and Destination Environment and connect them to the stretched network.
Perform a performance test using these VMs.
- Use one VM as an iperf server.
  - ```
  iperf3 -s -p <port>
```
- Use one VM as an iperf client.
  - ```
  iperf3 -c <iperf server IP> -p <port>
```

Additional Information

The following is an example of Windows VMs performing poorly compared to UPSA (Linux) VMs running on the same layer-2 stretched network.

(The values will be unique to each environment)

Windows VM performance test

PS D:\temp\Iperf> .\iperf3 -c <Dest VM> -p 88
 Connecting to host <Dest VM>, port 88
 [  4] local <Source VM IP> port 56442 connected to <Dest VM IP> port 88
 [ ID] Interval        Transfer  Bandwidth
 [  4]   0.00-1.00   sec  7.12 MBytes  59.8 Mbits/sec
 [  4]   1.00-2.00   sec  5.38 MBytes  44.9 Mbits/sec
 [  4]   2.00-3.00   sec  4.88 MBytes  40.9 Mbits/sec
 [  4]   3.00-4.00   sec  6.50 MBytes  54.5 Mbits/sec
 [  4]   4.00-5.00   sec  5.88 MBytes  49.3 Mbits/sec
 [  4]   5.00-6.00   sec  5.38 MBytes  45.1 Mbits/sec
 [  4]   6.00-7.00   sec  7.00 MBytes  58.7 Mbits/sec
 [  4]   7.00-8.00   sec  5.38 MBytes  45.1 Mbits/sec
 [  4]   8.00-9.00   sec  6.88 MBytes  57.7 Mbits/sec
 [  4]   9.00-10.00  sec  7.38 MBytes  61.8 Mbits/sec
 - - - - - - - - - - - - - - - - - - - - - - - - -
 [ ID] Interval        Transfer  Bandwidth
 [  4]   0.00-10.00  sec  61.8 MBytes  51.8 Mbits/sec               sender
 [  4]   0.00-10.00  sec  61.6 MBytes  51.7 Mbits/sec               receiver

UPSA VM performance test

root@<UPSA VM>:/root |> iperf3 -c -p 88 <Dest VM IP>
Connecting to host  <Dest VM IP>, port 88
[  5] local <Source VM IP> port 40932 connected to  <Dest VM IP> port 88
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  53.2 MBytes   446 Mbits/sec    0    884 KBytes       
[  5]   1.00-2.00   sec  59.4 MBytes   498 Mbits/sec    0    930 KBytes       
[  5]   2.00-3.00   sec  60.7 MBytes   509 Mbits/sec    0    976 KBytes       
[  5]   3.00-4.00   sec  61.0 MBytes   511 Mbits/sec    0   1022 KBytes       
[  5]   4.00-5.00   sec  65.8 MBytes   552 Mbits/sec    0   1.04 MBytes       
[  5]   5.00-6.00   sec  68.1 MBytes   571 Mbits/sec    0   1.09 MBytes       
[  5]   6.00-7.00   sec  71.2 MBytes   597 Mbits/sec    0   1.13 MBytes       
[  5]   7.00-8.00   sec  72.5 MBytes   608 Mbits/sec    0   1.17 MBytes       
[  5]   8.00-9.00   sec  76.2 MBytes   640 Mbits/sec    0   1.22 MBytes       
[  5]   9.00-10.00  sec  81.2 MBytes   682 Mbits/sec    0   1.33 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   669 MBytes   561 Mbits/sec    0             sender
[  5]   0.00-10.04  sec   667 MBytes   557 Mbits/sec                  receiver