End-to-End MTU Validation for VMware HCX and vSphere Networks

Products

• VMware HCX

Issue/Introduction

Purpose

This article provides a structured methodology to validate Maximum Transmission Unit (MTU) configurations across the entire network path in a VMware HCX and vSphere environment.  Consistent MTU settings are required end-to-end; if one segment does not support larger frames, all other segments in that path must be adjusted to match those restrictions.

Symptoms

• HCX migrations or synchronizations fail or stall across new setup environments.
• ESXi hosts intermittently fall offline following HCX Service Mesh deployment.
• Network interrupts observed during high-traffic migration tasks.

Environment

• HCX
• VMware vSphere ESXi

Cause

MTU mismatches in the virtual or physical network path cause packet fragmentation or drops. HCX requires a consistent MTU across the Service Mesh path to maintain stable tunnels.

Resolution

End-to-End Validation Procedure

Prerequisites

• Administrative access to the vCenter Server.
• Root access to ESXi hosts.
• SSH access to HCX Manager and IX appliances.
• Management access to physical network switches.

1.  Validate Physical Switch Configuration:
   1. Ensure all physical switch ports involved in the HCX Service Mesh path (uplinks, inter-switch links, and host-facing ports) are configured for the target MTU.
   2. Verify configuration for Jumbo Frames (typically 9000 bytes) if applicable.
   3. Ensure the MTU is consistent across all hop points in the path.
2. Validate vSphere Distributed Switch (DVS) MTU
   1. Navigate to the vCenter Networking tab.
   2. Select the target DVS > Configure > Properties.
   3. Confirm the MTU value matches your network architecture (e.g., 9000 for Jumbo Frames).
3. Check VMkernel Adapter MTU
   1. For each host in the cluster, go to Configure > Networking > VMkernel adapters.
   2. Select the adapters used for vMotion or Replication.
   3. Ensure the MTU matches the DVS configuration. If a mismatch exists, update the VMkernel adapter to align with the DVS.
4. Execute End-to-End Connectivity Test (vmkping)
   1. Use the vmkping utility from an ESXi host to perform a non-fragmented packet test across the network.
      Command:
      vmkping -d -s <PAYLOAD_SIZE> <DESTINATION_IP>
      • -d: Sets the "Do Not Fragment" (DF) bit, crucial for detecting fragmentation.
      • -s: Defines the payload size.
      • Note: The payload size is the MTU value minus 28 bytes (IP and ICMP headers).
      • For 9000 MTU, use 8972.
      • For 1500 MTU, use 1472.
      • Example (Jumbo Frames):
        vmkping -d -s 8972 192.168.10.5
      • Success: Indicates end-to-end path MTU consistency.
      • Failure: If this fails, perform a test with a smaller packet size (e.g., 1472). If smaller packets succeed, you have confirmed an MTU path bottleneck.
5. Validate HCX Appliance MTU
   1. SSH into the HCX IX appliance.
      1. Run
         ip link show
     or
         ifconfig
to inspect interface MTU settings.
      2. Verify the appliance MTU aligns with the underlying network segment configuration.

Additional Information

Troubleshooting

• Packet Drops: If connectivity tests fail with large packets but succeed with small packets, investigate physical switch port configurations or intermediate firewall/router settings.
• Migration Stalls: If migrations stall at specific percentages (e.g., 0% base sync), MTU issues are a high-probability root cause. Ensure all path segments support the configured MTU.

Additional References

• If you are in need of assistance, open a new case with Broadcom. See: Creating and managing Broadcom support cases
• To download the latest software, see Download Broadcom Products and Software