VMs over L2E Network are unable to connect to their GW

Products

• VMware HCX

Issue/Introduction

• Certain VMs on an L2E network keep loosing network connectivity.
• When the issue occurs this VM cannot get to the GW or northbound to internet.

  • L2 communication is unaffected.

• Utilizing QLogic 2x25GE with QEDENTV driver.
• ARP requests for the GW was confirmed traversing through BOTH NE switchports (Source/Destination). 
  • This can be done by following the steps in the following KB.
    • Performing Targeted Packet Captures on HCX Network Extension Switchports
• ARP request was also seen leaving the TX capture point of host physical uplink. In the following example we used uplink4.
  
  
  
• The same ARP requests are also seen looping back through the receive (RCV) capture point of same uplink (vmnic4).
  

Environment

VMware HCX

QLogic 2x25GE

Cause

• ARP request for GW is not being completed. 
• After further investigation we identified we are running a Qlogic 2x25GE nic with the 'qedentv' driver utilizing the NPAR function.
• This is a "known issue" with these adapters when NPAR is utilized.

Resolution

• The below steps should be performed to disable "NPAR TX switching" function of NIC.

1. Place ESX host in MM.
2. Disable NPAR TX switching:
   

   esxcfg-module -s 'npar_tx_switching=0' qedentv

3. Reboot ESXi host.
4. Once host is rebooted, wait for the host to show as "connected" in vCenter.
5. Remove from MM and vMotion the NE VM to host.
6. Repeat on remaining ESX hosts in cluster. 

Additional Information

• Multicast and broadcast traffic looping back to VMs when using DVS
• L2 loop causes invalid MAC learning on NSX-T Edge Node when NPAR TX switching and L2VPN is enabled
• HSRP IP with virtual mac becomes unreachable