Validate RDMA connectivity and evaluate its performance in ESX

Products

• VMware vSphere ESXi

Issue/Introduction

This article introduces the new built-in utility to validate RDMA connection and evaluate the performance in VCF 9.1.

• Previously, VCF there was not an easy way to check if the ESXi hosts could form an RDMA connection over RoCEv2.
  • This depends on the availability of RDMA capable NICs on the hosts.
• Starting from VCF 9.1, there is a new built-in utility (rdmaperf) that will enable the ability to check for this functionality so RDMA applications can be run between nodes.
  • It can also be used to evaluate the throughput, latency, and CPU cost data to get a better comparison between RDMA and traditional TCP/IP.

Environment

VMware ESXi 9.1.0.0 and later

Resolution

1. Select two hosts and create the VMKernel adapter backed by the RDMA adapter, following the guideline from Configure Remote Direct Memory Access Network Adapters. 
2. Start the server cmd in host1:
   
   [root@host1:~] vmkload_mod rdmaperf
Module rdmaperf loaded successfully

[root@host1:~] /usr/lib/vmware/rdma/bin/rdmaperf-uw -s -l <vmkernel_adapter_ip_server>

3. Start the client cmd in host2:
   
   [root@host2:~] vmkload_mod rdmaperf
Module rdmaperf loaded successfully

[root@host2:~] /usr/lib/vmware/rdma/bin/rdmaperf-uw  -c -r <vmkernel_adapter_ip_server> -l <vmkernel_adapter_ip_client>  -o write_bw -d 10 
   

   -----------------------------------------------------
Operation type:           WRITE_BW
Queue pair type:          RC
Local end-point address:  192.168.122.2
Remote end-point address: 192.168.122.1
Port number:              18888
Is client:                YES
Use RDMA CM:              YES
Queue size:               64
Post list size:           4
Number of connections:    1
Test duration:            10 seconds
Message size:             4096
-----------------------------------------------------

   --------------------------------
Test results for conn id: 1 
Message size:         4096 bytes
Iterations:           4284337
Avg:                  6693.86 MB/s
--------------------------------
Total avg BW of this endpoint is: 8693.86 MB/s
-----------------------------------------------------

4. Unload the module after experiments are done:
   
   [root@host1/2:~] vmkload_mod -u rdmaperf