Communication failure over HCX Layer 2 extended network due to destination Distributed Firewall policy

Products

• VMware HCX

Issue/Introduction

Network connectivity issues often occur after an HCX Layer 2 (L2) network extension, specifically where virtual machines at the destination site are unable to communicate with the gateway or on-premises resources. This article outlines steps to troubleshoot and resolve issues where the Distributed Firewall (DFW) at the destination site blocks traffic for the migrated virtual machine.

• Virtual machines cannot ping the gateway after migration to the destination site.
• Packet traces using pktcap-uw confirm traffic is dropped at the dvfilter (Distributed Firewall) layer.
  [root@ESXi:~] pktcap-uw --trace --ip <VM-IP> -c 2
The trace session is enabled.
...
Accept...
Vsock connection from port 1142 cid 2.
10:19:14.182047[1] PktHandleID: gG+PhcBFAAA=, Captured at PktFree point, TSO not enabled, Checksum not offloaded and not verified, SourcePort #######, QID 0, headroomlen 314, length 74.
        PATH:
          +- [timestamp] | VnicTx | ######### |
          +- [timestamp] | PortInput | ######### |
          +- [timestamp] | IOChain | | [email protected]#1.0.8.0.24866350
          +- [timestamp] | IOChain | | [email protected]#1.0.8.0.24866350
          +- [timestamp] | IOChain | | [email protected]#v2_13_0_0
          +- [timestamp] | PreDVFilter | |
          +- [timestamp] | PktFree | |
  
  
• Traffic successfully leaves the source interface but fails to leave the ESXi host uplink interface..

Cause

Virtual machine packets are being dropped by the destination Distributed Firewall (DFW) because of specified security rules.

Resolution

1. Identify the DFW filter name attached to the virtual machine on the ESXi host: summarize-dvfilter | grep -i <VM_Name> -A4
   

   world #### vmm0:DB02 vcUuid:'VM-UUID'
    port ##### <vmName.eth0>
     vNic slot 2
      name: <dvFilter-Name>

2. Query the filter statistics to confirm policy-based drops: vsipioctl getfilterstat -f <filter_name>
   

   PACKETS                       IN                OUT
   -------                       --                ---
   v4 pass:                      16530              14621
   v4 drop:                          0                725 <- This number increases.
   ...
   
   DROP REASON
   -----------
   match drop rule tx packets: 725 <- This number increases.
   

3. Identify the specific rule causing the discard within the NSX Manager UI under Security > Distributed Firewall.
4. Modify the security policy or group membership criteria at the destination site to allow the required traffic for the affected virtual machine.
5. If the issue persists, contact Broadcom Support: Contact Broadcom Support.