Troubleshooting Intermittent Connectivity on HCX Extended Networks

Products

• VMware HCX

Issue/Introduction

VMware HCX extended networks are experiencing intermittent connectivity loss, particularly affecting Red Hat Enterprise Linux (RHEL) VMs or workloads on shared Network Extension (NE) appliances. This article provides troubleshooting steps and workarounds to restore connectivity when symptoms include abrupt SSH or ping failure.

 

Symptoms

• Intermittent SSH and ping loss for VMs on HCX-extended segments.
• Connectivity restored immediately after triggering guest traffic (e.g., ping GW from guest).
• NE uplink lossRate alerts in appliance logs.
  • /common/logs/admin/app.log
    

    <timestamp> UTC [InterconnectService_SvcThread-301395, J:2143e097, , TxId: <UUID>] INFO  c.c.s.m.j.c.v.h.i.u.InterconnectUnderlayEventUtil- handleUnderlayEventAlerting: eventCode LOSS_EXCEEDED

• Physical NIC buffer overrun alerts on hosts supporting the NE appliance.

  • esxcli network nic stats get -n vmnic3

    
    VMNIC3
    

Environment

• VMware HCX 4.x
• VMware Cloud Foundation (VCF) / NSX-T

Cause

• Connectivity issues in shared NE environments often stem from high extension density, leading to physical NIC buffer overruns.
• Additionally, a known behavior occurs where VDR MAC entries age out on the IDLE VM's host but are retained by the NE appliance host, causing the NE ESX host to drop ARP requests (due to proxy-arp behavior) from IDLE VM's ESX host.

Resolution

To resolve this issue, apply the following configuration changes and architectural optimizations:

1. Reduce Extension Density: Limit HCX extended networks per NE appliance to prevent physical NIC buffer overruns.
2. Isolate Appliances: Avoid co-location of HCX appliances on hosts that also support high-throughput NSX Edges.
3. Dedicated NE Migration: For critical workloads experiencing persistent drops, migrate these segments to a dedicated NE appliance.
4. Apply Gateway Workaround: Implement a guest-level script within idle VMs to ping the default gateway every 5-10 minutes. This prevents the VDR MAC entries from aging out.

Additional Information

• HCX Network Extension Performance Tuning