Troubleshooting HCX Network Extension Datapath Failures
search cancel

Troubleshooting HCX Network Extension Datapath Failures

book

Article ID: 377288

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

HCX Network Extension service allows creation of a Layer 2 network at destination HCX site (utilizing NSX) and bridge this remote network to the source network. There are a variety of potential reasons why this Datapath could fail. This article will serve as a point of reference to review some possible causes for Datapath failure as well as what documentation is required when opening a support request with Broadcom. 

Environment

VMware HCX

Resolution

  1. Below are some common troubleshooting steps to perform:

    1. Source/on-prem VM on network A cannot communicate with cloud/target VM on stretched network A.
      1. Verify the IP config (subnet, gateway etc.) is correct inside guest-OS of each affected VM on cloud.
    2. On both HCX-MGRs UI verify the tunnel state.
      1. If the tunnel state shows "red" , check if NE (Network Extension) uplink interface can reach its respective HCX-MGR (NE-I with HCX-Connector etc..)
        1. Check connectivity between NE Uplink interfaces. 
          1. type "ccli".
          2. Type "list" to show HCX appliances.
          3. Type "go #" - replace # with the number of the appliance.
          4. Type "ssh" to change context to CLI of appliance.
          5. Ping the peer HCX-NE appliance uplink IP : "ping #.#.#.#" 
            1. If this is not successful - check that the NE networks are "connected" in VC UI. 


          6. Verify the UDP 4500 packets are leaving ESXi. To do this follow the steps below. 
            1. Find the MAC of NE uplink interface using VC UI. (see above screenshot)
            2. SSH to the esxi host the NE resides on. 
            3. run the command "netdbg vswitch instance list" - find the MAC corresponding to uplink interface. Record what vmnic it is using. 
            4. Perform a packet capture from ESXI 
              1. pktcap-uw --capture UplinkSndKernel,UplinkRcvKernel --uplink vmnic# -o - | tcpdump-uw -r - -ean host uplink-IP
                1. Pay attention to 2 types of traffic.
                  1. The ICMP traffic which was initiated via ping from NE directly.
                  2. The UDP 4500 packets NE sends to its peer to establish the tunnel. 
                2. If no ICMP reply or UDP traffic from peer NE is seen then this is an indication of a physical networking issue. Remember this trace shows packets are leaving the physical ESXi NIC and therefore handed off to the physical network. 
                3. Type "ctrl-c" to stop the capture. 

    2. Alternative method to find the NE appliance Uplink Interface:

    1. Access the HCX Manager UI.
      1. Navigate to the Service Mesh and view the appliances.
      2. Make note of the IP address that is associated with the Uplink.
    2. SSH into the HCX Manager.
      1. Navigate to the NE appliance (ccli > list > go ID# associated with NE appliance > ssh).
      2. Issue the 'ifconfig' command.
      3. Find the interface that has the IP address from the above step.
        2. Make note of the MAC Address.
      4. Navigate to the vCenter and locate the NE Appliance VM.
      5. Edit settings on the VM.
      6. Expand the network adapters and locate which one has the same MAC Address.


    3. Navigate to Services - Appliances - Select and expand the NE appliances - open  Network Extension Details and select the extended network, select Show More Details
         1. Verify the NE appliance can learn Local MAC addresses
             
         2. Migrate a VM to the same host as the NE appliance and check if the Local MACs are populating.
         3. Investigate further.

    4. This interface can now be used when reviewing ESXi Top output to identify the VMNIC that is used to hand off traffic to the physical environment.

    If the above steps show connectivity is healthy, however, there are still connection issues being reported by HCX UI, please file a request with Broadcom support and provide the information below. 

      1. This KB ID.
      2. HCX version.
      3. Brief problem description.
        1. Include Source VM name, IP/subnet and network segment/portgroup.
        2. Include Destination VM name, IP/subnet and network segment.
        3. OS type/version for affected VM's used for testing.
        4. VMware tools version.
      4. Is MON (Mobility Optimized Networking) enabled?
      5. What NE # is servicing this extension

 

Additional Information

Helpful documents that should be reviewed to ensure HCX NE(Network Extension) configuration is valid:

MON (Mobility Optimized Networking) is a feature of HCX Network Extension service that allows a cloud/destination VM on segment A to communicate with another cloud VM on segment B without having to route back to on-prem/source gateway. 

HCX NE appliances can be configured for High Availability (HA). This further protects extended networks from a NE appliance failure at either site.

Known issues with HCX Network Extension service:

Powered by Wolken Software