Capture RDMA packets in ESX

Products

• VMware vSphere ESXi
• VMware Cloud Foundation

Issue/Introduction

This article introduces new RDMA packet capture support in ESX and provides the details on how to use it.

VCF 9.1 simplifies capturing of RDMA packets for troubleshooting and debugging purposes using the RDMA Sniffer capability.

Environment

VMware ESX 9.1.0.0 and above

Resolution

1. Check if the used RMDA adapter supports the "RDMA Sniffer" feature:
   
   [root@host1:~] esxcli rdma device sniffer get
RDMA Device  Sniffer status
-----------  --------------
vmrdma0      N/A
...
vmrdma4      off
   
   "N/A" means no support.
   "off" means support, but disabled.
   "on" means support and enabled.
   
   
2. Identify the paired uplink adapter:
   
   [root@host1:~] esxcli rdma device list
Name     Driver      State    MTU  Speed     Paired Uplink  Description
-------  ----------  ------  ----  --------  -------------  -----------
vmrdma0  bnxtroce    Active  1024  25 Gbps   vmnic0         Broadcom RoCE device
...
vmrdma4  nmlx5_rdma  Active  1024  100 Gbps  vmnic4         MT43244 Family  [BlueField-3 integrated ConnectX-7]
   
   
3. If the used RDMA adapter is vmrdma4 which has the sniffer feature support, we can go ahead with the following process.
   1. Enable the sniffer:
      
      esxcli rdma device sniffer set -e 1 -d vmrdma4
      
      
   2. You will be able to check the sniffer status for this device:
      
      [root@host1:~] esxcli rdma device sniffer  get -d vmrdma4
RDMA Device  Sniffer status
-----------  --------------
vmrdma4      on
      
      
   3. Run some RDMA workload in ESX(e.g. VSANoRDMA, NVMeoRDMA, ISER, rdmaperf-uw tool).
   4. Capture RDMA packet:
      
      pktcap-uw --capture RDMASniffer --uplink vmnic4(picked from above paired uplink)  -c 1000 -o /tmp/sniffer.pcap --dstudpport 4791
      
      
   5. After the capture session is completed, don't forget to disable the sniffer:
      
      esxcli rdma device sniffer set -e 0  -d vmrdma4