How to use iPerf to test bandwidth between source host and target vSphere Replication appliance

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Source host replicates data to the target replication appliance w.r.t the RPO set on a VM. iPerf can help us in simulating the same path replication traffic takes.

Resolution

1. Disable ESXi firewall on the source ESXi host that contains the most number of replicated VMs and the target host that contains the VR (vSphere Replication Appliance) 

2. Run this command to disable firewall on ESXi - esxcli network firewall set --enabled false

   [root@ESXi67APR:/usr/lib/vmware/vsan/bin] esxcli network firewall get
   Default Action: DROP
   Enabled: false
   Loaded: true
 

Now its time to run a test to find out the actual bandwidth between the source and target server to calculate the precise bandwidth available for vSphere Replication traffic. Please note that vSphere Replication, vSphere Replication NFC and Management traffic can lie on different VMKernel adapters depending on the your network configuration. Please refer to Isolating the Network Traffic of vSphere Replication (78613) to get a better understanding of this.

Find out which VMK adapter is enabled to use vSphere replication traffic on the source host. I would recommend that you also ensure that this configuration is uniform across the cluster/datacenter otherwise it could cause replication performance anomalies. Checking this is key to running the iPerf test because if the vSphere replication traffic service is enabled on a different VMK but you are using the VMK IP address belonging to management VMK, based on the NICs used for Management (1Gbps) & vSphere Replication (10Gbps) you could see a totally different result. Even if they both are using the same capacity NIC cards but are on different subnets, it would still be a very good choice to use the VMK IP address belonging to vSphere Replication traffic type because this will help us in determining if there’s any network congestion/latency on this subnet.
 

Iptables is a firewall built for Linux operating systems, its used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Its a command-line firewall utility that uses policy chains to allow or block traffic. Use the command below to find out if a port/s is in ACCEPT state before starting an iPerf test. 

Accept – Allow the connection.
Drop   – Drop the connection. 
Reject – Don’t allow the connection, but send back an error. 

Iptables is a firewall built for Linux operating systems, its used to set up, maintain, and inspect the tables of IP packet filter rules in the Linux kernel. Its a command-line firewall utility that uses policy chains to allow or block traffic. Use the command below to find out if a port/s is in ACCEPT state before starting an iPerf test. 

Accept – Allow the connection.
Drop   – Drop the connection. 
Reject – Don’t allow the connection, but send back an error. 



Since, Port # : 31031 & 32032 are already in use by vSphere Replication Appliance, you will have to use a custom port for testing. You can use the command here to add a port to the iptables. 

iptables -A INPUT -p tcp --dport 5201 -j ACCEPT

dport > Sets the destination port for the packet.

Now, run the following command below to check if this port is added. 


 
Now, with the source ESXi host as the iPerf client & the target replication appliance as the iPerf server, you can run the tests as shown below. Use the --bind option to specify the IP of the replication network you want IPerf to utilize during testing to get accurate results. 

iperf3 -V --server --port 5201 --bind 192.168.50.7 -fM
Specify the IP Address for Incoming Storage Traffic on the target replication appliance 



./iperf3 -V --client 192.168.50.2 --port 5201 -fM
Specify the IP Address for Incoming Storage Traffic on the target replication appliance 
 

To interrupt an iPerf test, press Ctrl+c.

NOTE: HBR (Host Based Replication) uses single TCP stream per replicated VM, there's no way to enable multiple TCP streams per VM replication yet. As a result, the iperf test above reports low bandwidth for a single TCP stream. We can get closer to a realistic test by including more ESXi hosts as clients or running iPerf with parallel streams. 

To simulate a real-time scenario where multiple VM replications are spread across multiple hosts, you need to start multiple iPerf transfers concurrently. We can do this in 2 ways : 

1. Use more ESXi hosts as clients 
OR
2. Run iPerf using parallel streams

1. Using more ESXi hosts as clients

If you'd like to use multiple ESXi hosts, you need to run commands to accept more ports on the VR Appliance iptables firewall to scale with the number of hosts & also bind them with the target replication traffic IP address. 

Example :

iptables -A INPUT -p tcp --dport 5202 -j ACCEPT
iptables -A INPUT -p tcp --dport 5203 -j ACCEPT

iperf3 --server --port 5202 --bind 192.168.X.X
iperf3 --server --port 5203 --bind 192.168.X.X 

Now, start iPerf on these additional ESXi hosts and the VR appliance using multiple SSH sessions. The sum total of all the bitrate/bandwidth you are seeing on all the replication appliances on the target site is the bandwidth available between sites

2. Run IPerf using parallel streams 

Parallel Streams are multiple TCP connections opened by an application to increase performance and maximize the throughput between communicating hosts. With parallel streams, data blocks for a single file transmitted from a sender to a receiver are distributed over multiple streams.

In this section, parallel streams are specified by the client when executing the throughput test in iPerf. The iPerf server should start as usual, without specifying any additional options or parameters.

Launch iPerf in server mode on the target VR Appliance, run the command - iperf3 -V --server --port 5201 --bind 192.168.50.7 -fM



Launch iPerf in client mode from source ESXi host with the -P option specified (not to be confused with the -p option which specifies the listening port number). This option specifies the number of parallel streams.

./iperf3 -V --client 192.168.50.7 --port 5201 -P 10 -fM



./iperf3 -V --client 192.168.50.7 --port 5201 -t 30 -P 10 -fM 



-t indicates the time iperf runs for. The above command uses 10 parallel streams. Note that 10 sockets are now opened on different local ports, and their streams are connected to the server, ready for transmitting data and performing the throughput test. With this test, we can see a substantial rise in throughput rate of close to 2.29GB in contrast to the previous tests we have run before.

NOTE: It is important to note that, during the test, iPerf uses all available bandwidth of the communication channel between the client and server, which can negatively affect production application and users.

The following factors have a negative impact on replication speeds - 

1. Firewalls 
2. Improper routes set to reach the target site 
3. Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) monitoring network traffic 
4. Quality of Service (QoS) configurations set on port # 31031 & 32032 
5. WAN link used for various other traffic types that causes congestion thus impacting vSphere Replication traffic.

Additional Information

iPerf has a server/client architecture, in order to test a network connection between two systems, you will need to have iPerf on both systems. iPerf is a command-line tool used in diagnosing network speed issues by measuring the maximum network throughput a server can handle. It is particularly useful when experiencing network speed issues, as you can use iPerf to determine which server is unable to reach maximum throughput.

iPerf - The ultimate speed test tool for TCP, UDP and SCTP - https://iperf.fr/iperf-doc.php

vSphere Replication Bandwidth Calculator - https://core.vmware.com/?share=web_tool2458
 

-s	Run in server mode
-i	Time between periodic bandwidth reports, in seconds /  display transfer data every 2 seconds instead of the default of 10 (-i 2)
-t	time iperf will be running / Length of time to transmit, in seconds / we want the test to run for 20 seconds instead of 10
-c	client ip, will force the usage of the correct VMkernel interface
-V, --verbose	give more detailed output
-fM	Format, 'M' = MBytes/sec
-B, --bind host	Bind to host, one of this machine's addresses. For the client this sets the outbound interface. For a server this sets the incoming interface. This is only useful on multihomed hosts, which have multiple network interfaces. It also defines the IP the iperf server will listen to