Using iPerf for Testing vSAN Network Performance

Products

• VMware vSAN

Issue/Introduction

vSAN strongly depends on the performance of the underlying network for the resulting performance of vSAN storage.

For troubleshooting purposes it may be necessary to validate the bandwidth available for use on the vSAN network. ESXi supports the use of the iperf utility for this testing, as explained below.

Environment

VMware vSAN

Resolution

• Step 1: Locate the iperf utility and grant necessary permissions 
  
  On an esxi host enter the following to change to the working directory for iperf.
  
  cd /usr/lib/vmware/vsan/bin

Looking in this folder, you may see multiple copies of the iperf application. It may be named iperf3 or iperf, depending on the version. Versions of ESXi prior to 7.0u3 prevented running this command directly and it was necessary to make a copy of the iperf program by running the following command:
  
  cp /usr/lib/vmware/vsan/bin/iperf3 /usr/lib/vmware/vsan/bin/iperf.copy2

On ESXi 8.0 and above, it is not necessary to make a copy of the program using the command above. You may see the error shown below, and the earlier method of creating an iperf3.copy and using execInstalledOnly might not work.

  /usr/lib/vmware/vsan/bin/iperf3 -s
iperf3: error - unable to start listener for connections: Operation not permitted

  To make it work we will have to disable the security policy which handles app execution called "appDom". This setting can be checked as follows:

  localcli system secpolicy domain list | grep appDom
appDom                   enforcing

  Change the enforcement level for the Domain "appDom" from "enforcing" to "disabled":

  localcli system secpolicy domain set -n appDom -l disabled
  
  

 

• Step 2: Disable the ESXi Firewall on all hosts being tested

         esxcli network firewall set --enabled false

 

• Step 3: Start iPerf in Server Mode on one host
  
  Use the command iperf with -s for server. Optionally, the bind IP address can be specified with -B <ip address> (case sensitive), the server port can be specified with -p, and the reporting interval in seconds can be specified with -i. The default port is 5201.
  
  The output will appear similar to this:

         [root@Example_Server1:/usr/lib/vmware/vsan/bin] ./iperf3 -s -B 192.168.0.1 -p 5201 -i 1
         ------------------------------------------------------------
        Server listening on TCP port 5201
        Binding to local address 192.168.0.1
        TCP window size: 64.0 KByte (default)
        ------------------------------------------------------------
        [ 4] local 192.168.0.1 port 5201 connected with 192.168.0.2 port 65535
        [ ID] Interval Transfer Bandwidth
        [ 4] 0.0- 1.0 sec 866 MBytes 7.27 Gbits/sec
        [ 4] 1.0- 2.0 sec 937 MBytes 7.86 Gbits/sec
        [ 4] 2.0- 3.0 sec 921 MBytes 7.72 Gbits/sec
        ...

 

• Step 4: Run iPerf in client mode and connect to the server
  
  Use the command iperf with -c for client followed by the IP address of the server in the previous step. The target port can be specified with -p.
  
  The results will look like this:
  
  

        [root@Example_Server2:/usr/lib/vmware/vsan/bin] ./iperf3 -c 192.168.0.1
        ------------------------------------------------------------
        Client connecting to 192.168.0.1, TCP port 5201
        TCP window size: 0.03 MByte (default)
        ------------------------------------------------------------
        [ 5] local 192.168.0.2 port 65535 connected with 192.168.0.1 port 5201
        [ ID] Interval Transfer Bandwidth
        [ 5] 0.0-10.0 sec 950 MBytes 1.1 Gbits/sec
       ...

 

• Step 5: Re-enable the ESXi Firewall on all hosts
  
  To re-enable the firewall, use the command
  esxcli network firewall set --enabled true
  
  To re-enable the security policy, use the command
  esxcli system secpolicy domain set -n appDom -l enforcing

       

 

 

Additional Information

Refer to the article vSAN Networking – Network Oversubscription for a detailed discussion on the relationship between network capacity and vSAN.