Performing Targeted Packet Captures on HCX Network Extension Switchports

Products

VMware HCX

Issue/Introduction

When unexpected packet loss or intermittent connectivity issues occur on HCX Network Extension VLANs between sites, administrators need to identify the exact cause of the disruption.
These intermittent issues are difficult to diagnose without proper packet capture evidence from the specific switchport experiencing the problem.
No clear error message appears in the HCX Manager UI during these events, making it necessary to gather packet-level details.
Pinpointing the exact VLAN experiencing packet loss and capturing the relevant traffic requires a systematic approach.
This article explains how to perform targeted packet captures when troubleshooting connectivity issues on HCX Network Extensions.

Environment

VMware HCX
VMware vSphere ESXi
VMware vCenter Server

Cause

Packet loss and intermittent connectivity on Network Extensions can be caused by various factors that are only identifiable through detailed packet analysis:

Physical network infrastructure issues between sites
Resource constraints on the ESXi host running the Network Extension appliance
Firewall or security policy blocking specific traffic
Network congestion during peak usage periods
Hardware-related issues on networking equipment
Configuration of firewalls or switches in infrastructure outside of the ESXi host

Packet captures can help make it easier to determine which of these factors is causing the intermittent connectivity issues on specific VLANs.

Resolution

1. Identify the Specific Switchport for the Affected VLAN

a. Connect to your ESXi host via SSH.

b. Find your HCX Network Extension appliance's World ID:

esxcli network vm list | grep HCX

c. Map the Network Extension's switchports to their corresponding VLANs by following the detailed correlation process in KB 388893: Correlating HCX Network Extension Switchports to VLANs

d. Document the switchport ID for the VLAN experiencing issues:

VLAN ID: 222

Switchport ID: 78453###

MAC Address: 00:50:56:00:74:22

2. Set Up a Targeted Packet Capture

a. Connect to the ESXi host via SSH and navigate to a suitable datastore for storing captures:

cd /vmfs/volumes/<datastore_name>

mkdir hcx_packet_captures

b. Execute a focused packet capture on the specific switchport:

pktcap-uw --switchport <switchport_ID> -o /vmfs/volumes/<datastore_name>/hcx_packet_captures/vlan_capture.pcap

c. For detailed information about additional pktcap-uw options, refer to KB 341568: Packet capture on ESXi using the pktcap-uw tool.

3. Capture at Multiple Points for Complete Analysis

a. Capture at the VM switchport level to see traffic from the HCX Network Extension appliance:

pktcap-uw --switchport <switchport_ID> --capture VnicTx,VnicRx -o /vmfs/volumes/<datastore>/switchport_capture.pcap

b. Simultaneously capture at the physical uplink to determine if packets are being dropped between the virtual and physical layers:

pktcap-uw --uplink vmnic<#> --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/<datastore>/uplink_capture.pcap

c. Add filtering to focus on relevant traffic:

# Only capture traffic to/from a specific endpoint

pktcap-uw --switchport <switchport_ID> --ip 10.#.#.# -o /vmfs/volumes/<datastore>/filtered_capture.pcap

# Limit packet size to just capture headers (reduces file size)

pktcap-uw --switchport <switchport_ID> -s 128 -o /vmfs/volumes/<datastore>/headers_only.pcap

# Capture only a specific protocol (e.g., ICMP)

pktcap-uw --switchport <switchport_ID> --proto 0x01 -o /vmfs/volumes/<datastore>/icmp_capture.pcap

4. Analyze the Captured Packets for Signs of Network Issues

Download the capture files and analyze them in Wireshark, looking specifically for:

a. TCP retransmissions indicating packet loss:

Filter: tcp.analysis.retransmission
What to look for: Clusters of retransmissions or patterns tied to specific timing or endpoints

b. Round-trip time anomalies showing latency issues:

Filter: tcp.analysis.ack_rtt
What to look for: Sudden spikes in RTT values or consistently high RTT to certain destinations

c. Duplicate ACKs signaling packet loss:

Filter: tcp.analysis.duplicate_ack
What to look for: Three or more consecutive duplicate ACKs (triggers fast retransmit)

d. Zero window conditions indicating buffer issues:

Filter: tcp.analysis.zero_window
What to look for: Periods where the receiving host cannot accept more data

e. TCP errors and connection resets:

Filter: tcp.flags.reset == 1
What to look for: Unexpected connection terminations

5. Correlate with System Performance Metrics

While the packet capture is running, collect system performance data:

a. Run esxtop to monitor real-time network statistics:

# Press 'n' after starting esxtop to view network statistics

esxtop

b. Look specifically for these indicators:

DRPTX% and DRPRX% (dropped packets transmitted/received)
PKTTX/s and PKTRX/s (packets transmitted/received per second)
MbTX/s and MbRX/s (megabits transmitted/received per second)

c. Take screenshots of esxtop output at regular intervals or when users report issues.

6. Apply Targeted Fixes Based on Findings

a. If you see packet loss at the virtual switchport but not at the physical uplink:

Run simultaneous packet captures on both the switchport and physical uplink:

# Capture at both points at the same time

pktcap-uw --switchport <switchport_ID> -o /vmfs/volumes/<datastore>/switchport_capture.pcap &
pktcap-uw --uplink vmnic<#> -o /vmfs/volumes/<datastore>/uplink_capture.pcap &

Compare the two captures for differences to identify where packets are being dropped
Look for packets present in one capture but missing in the other
Check for resource contention on the ESXi host
Verify the HCX Network Extension appliance has adequate resources

b. If you see packet loss at both virtual and physical layers:

Work with network administrators to investigate physical network issues
Verify firewall and security policies aren't interfering with traffic

c. Important note on packet loss diagnosis:

Packet loss is not always caused by the HCX Network Extension itself be sure to find the source and destination addresses in the captures. Work with network team after identifying the traffic to be sure this is not a misconfiguration or issue in configuration within the physical infrastructure
If analysis shows high CPU contention on the HCX Network Extension appliance, refer to the performance tuning recommendations in KB 389281
If resource utilization appears normal, the issue is likely external to HCX and should be diagnosed based on traffic patterns observed in the packet captures

e. If you see intermittent complete traffic disruption:

Review network infrastructure between sites
Check for failover or redundancy issues
Verify network hardware functioning properly

7. Implement Performance Tuning for HCX Network Extensions

If your packet capture analysis reveals performance-related issues, refer to KB 389281: HCX Network Extension Performance Tuning for implementing performance optimization techniques, including:

a. Enabling Generic Receive Offload (GRO) to improve inbound traffic performance

b. Configuring Application Path Resiliency (APR) to create multiple transport tunnels

c. Enabling TCP MSS Clamping to optimize transport performance

d. Optimizing CPU Thread Allocation for high-density environments

e. Configuring Network Extension Appliance Scale Out for high-traffic environments

Additional Information

Document Findings for Support

If escalating to Broadcom Support is necessary:

a. Provide the following evidence:

The VLAN-to-switchport correlation information
All packet captures (both normal baseline and problem captures)
Screenshots of esxtop output during the issue
Timeline of when issues occurred
Any patterns identified in your analysis

b. Include:

Source and Target HCX log bundles
Screenshots of any error messages
Configuration details of the affected network extension
Packet captures and esxtop screenshots collected during an active incident