Collecting network packets using the lightweight PacketCapture on ESXi.

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

Describes the PacketCapture tool and steps to efficiently collect network packets on ESXi.

Environment

VMware vSphere ESXi 8.0.1

Resolution

Please follow the below steps for enabling the PacketCapture:

1. SSH to the VMware ESXi host..

2. Enable PacketCapture in the rhttpproxy config.
Note: Since ESXi 8.0U1 the service configuration is stored in a special config store database and is accessible using configstorecli tool.
To edit the configuration for previous releases, refer to: 52843

• Run the command to create a temporary JSON file containing rhttpproxy config options: 

      /bin/configstorecli config current get -c esx -g services -k rhttpproxy -outfile tmp.json

• Run the command to edit the file:

     /bin/vi tmp.json

• Add the following configuration options to the temporary JSON file:

{
   "packet_capture" : {
      "enabled": true,
      "validity": 2,
      "directory": "/var/run/log",
      "max_data_in_pcap_file": 52428800,
      "max_pcap_files_count": 5,
      "interface_list": [
         {
            "id": "lo0",
            "name": "lo0",
            "max_data_in_pcap_file": 52428800,
            "max_pcap_files_count": 5
         },
         {
            "id": "vmk0",
            "name": "vmk0",
            "max_data_in_pcap_file": 52428800,
            "max_pcap_files_count": 5
         }
      ]
   }
}

Description of the options in the Packet Capture configuration:

Enabled - Whether the tool is enabled or not.
Validity - On startup delete all pcap and pcap.gz files that were last modified more than X hours ago and are not part of the current process.
Directory - Controls where rolling pcap files are stored. The directory must exist and should be accessible.
Max_data_in_pcap_file - Size (in bytes) of captured data of each pcap file before rolling over to the next.
Storing 50 MB of captured data in a pcap file requires a pcap file of about~67.5 MB, the minimum amount on ESXi is 2.5MB.
Max_pcap_files_count - Number of pcap files to rotate amongst. The minimum number is 2.
Interface_list - Interfaces on which rhttpproxy captures packets. It could overwrite each of the PacketCapture's options configured in the config.

• Run the command to apply the file to the Database:

    /bin/configstorecli config current set -c esx -g services -k rhttpproxy  -infile tmp.json

• Run the command to restart rhttpproxy service:

    /etc/init.d/rhttpproxy restart

3. You should now see a new .pcap file in the folder /tmp (or if you configured a different destination folder in step 2 by modifying the directory field).

4. When the traffic is sniffed during the network problem, disable the PacketCapture. To disable the PacketCapture:

• Run the command to create a temporary JSON file containing rhttpproxy options:

/bin/configstorecli config current get -c esx -g services -k rhttpproxy -outfile tmp.json

• Run the command to edit the file:

/bin/vi tmp.json

• Edit the enabled field to false:

{
   "packet_capture" : {
      "enabled": false
   }
}

• Run the command to apply the file to the Database:

/bin/configstorecli config current set -c esx -g services -k rhttpproxy -infile tmp.json

• Run the command to restart rhttpproxy service:

/etc/init.d/rhttpproxy restart

5. Collect the support bundle - Collect the created pcap and pcap.gz files. In case directory was not provided, they are produced by default in /var/run/log.

6. Copy the pcap and pcap.gz files to a system that runs a network analyzer tool, such as Wireshark and examine the packet details.

Note: Before analyzing the ESXi packet details, fix frame size meta data as ESXi packets are truncated by design. To fix frame size, use a third-party tool such as TraceWrangler.

• Open the pcap files using the tool.
• Edit files by selecting "Fix frame size meta data" and "Also recalculate wire size when not hard sliced" under Packet list option.
• Click on Run to save the new files.
• Analyze the network using the newly generated pcap files.

NOTE: In vSphere 8.0U3, Capture of network packets by using the PacketCapture tool on ESXi does not work. This is a known issue. See VMware ESXi 8.0 Update 3 release notes To workaround this use pktcap-uw tool. For more information  see Capture and Trace Network Packets by Using the pktcap-uw Utility

Disclaimer: Broadcom is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that Broadcom ,see endorses, recommends, or accepts any responsibility for the content of such sites.

 

Additional Information

Impact/Risks:
PacketCapture collects and evaluate big amount of data which may consume a high CPU and disk usage.