Determining whether NSX Distributed Firewall (DFW) IPFIX Data is sent correctly to Aria Operations for Networks (AON/vRNI)
search cancel

Determining whether NSX Distributed Firewall (DFW) IPFIX Data is sent correctly to Aria Operations for Networks (AON/vRNI)

book

Article ID: 422758

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention VCF Operations for Networks

Issue/Introduction

  • In some scenarios, Aria Operations for Networks does not display Distributed Firewall IPFIX flow data, or the data appears incomplete or incorrect.
  • When this occurs, it is necessary to determine whether IPFIX data is being generated and transmitted correctly from the ESXi host where the virtual machine generating the traffic resides.
  • Packet captures taken directly on the ESXi host are the most reliable method to validate IPFIX transmission.

Environment

VMware NSX

Aria Operations for Networks (AON), formerly vRealize Network Insight (vRNI)

Resolution

The below procedure can be followed to confirm if Distributed Firewall IPFIX data is being generated and transmitted correctly from NSX to Aria Operations for Networks.
The high-level steps are:

  1. Run packet capture on ESXi host uplink
  2. Confirm IPFIX template is seen in the packet capture
  3. Identify the IPFIX Flow records in the packet capture
  4. Validate the IPFIX field values


Procedure:

  1. Run packet captures on the ESXi host uplinks where the VM generating the flow resides. Refer to this KB for running the pcaps - https://knowledge.broadcom.com/external/article/341568/using-the-pktcapuw-tool-in-esxi-55-and-l.html
    • IPFIX data is sent to the collector IP over the port configured in NSX. This is typically over UDP port 2055. NOTE: This collector configuration in NSX is automatically created when enabling DFW IPFIX on the NSX datasource within Aria Ops for Networks. 
    • Filter your packet capture to only capture traffic to the collector IP over UDP 2055 (or the port configured for the collector in the NSX UI under 'Plan & Troubleshoot > IPFIX > Collectors'). 

      Syntax:
      pktcap-uw --uplink <ESXi vmnic> --ip <collector IP> --udpport 2055 --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/<datastore>/<capture name>

      Variables:
      ESXi vmnic - This is the host uplink used by the VM generating the flow 
      collector IP - This is the AON collector IP 
      datastore - This is the datastore where you want to save the pcap filter 
      capture name - This is the name you give to the pcap file. Example: ipfix_data.pcap

      Example Capture:
      pktcap-uw --uplink vmnic2 --ip 10.X.X.10 --udpport 2055 --capture UplinkSndKernel,UplinkRcvKernel -o /vmfs/volumes/vm_ds01/ipfix_data.pcap

    • Run the packet capture for at least 5 minutes. This allows enough time to capture the IPFIX templates. 

  2. Confirm the IPFIX Template is seen in the packet capture. IPFIX templates define the structure of flow records and are required for both the collector and Wireshark to correctly parse IPFIX data. If the template is missing, flow records cannot be interpreted.
    • Open the packet capture in Wireshark
    • Confirm that an IPFIX template has been captured. For Distributed Firewall IPv4 flows, the required template is Template ID 294.
    • Use the following display filter:
      cflow.template_id == 294
      Example template packet:
      2025-12-05 18:29:06.398291  10.X.X.15  10.X.X.10  CFLOW  53527  2055  350 IPFIX flow (308 bytes) Obs-Domain-ID=38416 [Data-Template:294,295


      10.X.X.15 = ESXi host MGMT VMK IP
      10.X.X.10 = Collector IP

      Template 294 is for IPv4
      Template 295 is for IPv6

    • If the IPFIX template 294 is not captured, the the flow data will be displayed as "Malformed." Reference KB - Wireshark show IPFIX packet as "Group: Malformed"
    • If the IPFIX template 294 is not seen in the packet capture, run the packet capture again with a longer duration. 

  3.  After confirming the IPv4 template is present, identify IPFIX flow records.
    • Common display filters:

      Filter by the source VM IP address: 
      cflow.srcaddr == x.x.x.x   

      Filter by the DFW rule ID converted to hex format: 
      cflow.enterprise_private_entry == 00:0f:xx:xx 

      Example of a Wireshark display filter that filters for IPFIX flow packets that have a VM source IP of 10.X.X.30 and a rule ID of 1000010 (which converts to 00:0F:42:4A)
      cflow.flowset_id == 294 && cflow.enterprise_private_entry == 00:0F:42:4A && cflow.srcaddr == 10.X.X.30
  4. Once an IPFIX flow packet is identified, validate the IPFIX field values to confirm expected values are present
    • The authoritative list of fields for Distributed Firewall IPv4 IPFIX flows is documented here: ESXi Distributed Firewall IPFIX Templates 

      IPv4 template:

      IPFIX_TEMPLATE_FIELD(sourceIPv4Address,4)
      IPFIX_TEMPLATE_FIELD(destinationIPv4Address,4)
      IPFIX_TEMPLATE_FIELD(sourceTransportPort,2)
      IPFIX_TEMPLATE_FIELD(destinationTransportPort,2)
      IPFIX_TEMPLATE_FIELD(protocolIdentifier,1)
      IPFIX_TEMPLATE_FIELD(icmpTypeIPv4,1)
      IPFIX_TEMPLATE_FIELD(icmpCodeIPv4,1)
      IPFIX_TEMPLATE_FIELD(flowStartSeconds,4)
      IPFIX_TEMPLATE_FIELD(flowEndSeconds,4)
      IPFIX_TEMPLATE_FIELD(octetDeltaCount,8)
      IPFIX_TEMPLATE_FIELD(packetDeltaCount,8)
      IPFIX_TEMPLATE_FIELD(firewallEvent,1)
      IPFIX_TEMPLATE_FIELD(flowDirection,1)
      IPFIX_TEMPLATE_FIELD(ruleId,4)
      IPFIX_TEMPLATE_FIELD(sessionFlags,1)
      IPFIX_TEMPLATE_FIELD(reportingRole,1)
      IPFIX_TEMPLATE_FIELD(flowDirection,1)
      IPFIX_TEMPLATE_FIELD(flowId,8)
      IPFIX_TEMPLATE_FIELD(algControlFlowId,8)
      IPFIX_TEMPLATE_FIELD(algType,1)
      IPFIX_TEMPLATE_FIELD(algFlowType,1)
      IPFIX_TEMPLATE_FIELD(averageLatency,4)
      IPFIX_TEMPLATE_FIELD(retransmissionCount,4)
      IPFIX_TEMPLATE_FIELD(vifUuid,16)
      IPFIX_TEMPLATE_FIELD(vifId,48)
    • Here is how the fields are displayed in the packet captures. Expand the "Cisco Netflow/IPFIX" section within the packet to display the fields. Notice some of the fields are labeled "Enterprise Private entry":
      Flow 1

          SrcAddr: 10.X.X.30

          DstAddr: x.x.x.x

          SrcPort: 138

          DstPort: 138

          Protocol: UDP (17)

          IPv4 ICMP Type: 0

          IPv4 ICMP Code: 0

          [Duration: 0.000000000 seconds (seconds)]

          Octets: 229

          Packets: 1

          Firewall Event: Flow denied (3)

          Direction: Egress (1)

          Enterprise Private entry: (VMware Inc.) Type 950: Value (hex bytes): 00 0f 42 4A

          Enterprise Private entry: (VMware Inc.) Type 953: Value (hex bytes): 00

          Enterprise Private entry: (VMware Inc.) Type 954: Value (hex bytes): 01

          Flow Id: 2327545914962936460

          Enterprise Private entry: (VMware Inc.) Type 955: Value (hex bytes): 00 00 00 00 00 00 00 00

          Enterprise Private entry: (VMware Inc.) Type 956: Value (hex bytes): 00

          Enterprise Private entry: (VMware Inc.) Type 957: Value (hex bytes): 00

          Enterprise Private entry: (VMware Inc.) Type 958: Value (hex bytes): 00 00 00 00

          Enterprise Private entry: (VMware Inc.) Type 962: Value (hex bytes): 00 00 00 00

          Enterprise Private entry: (VMware Inc.) Type 960: Value (hex bytes): XX XX XX XX XX XX XX XX XX XX 00 00 00 00 00 00

          Enterprise Private entry: (VMware Inc.) Type 961: Value (hex bytes): XX XX XX XX XX XX XX XX XX XX 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    • The Enterprise Private fields appear in the same order as defined in the IPv4 template. This ordering allows identification of each field even when Wireshark labels them generically as "Enterprise Private Entry"
    • All Enterprise Private values are displayed in hexadecimal and must be converted to decimal for interpretation.

      Examples:
      Enterprise Private entry: (VMware Inc.) Type 950: Value (hex bytes): 00 0f 42 4A     <------- This is the rule ID in hex format

      Enterprise Private entry: (VMware Inc.) Type 953: Value (hex bytes): 00              <------- This is the session flags field displayed in hex format

      Enterprise Private entry: (VMware Inc.) Type 954: Value (hex bytes): 01              <------- This is the reporting role field displayed in hex format

  5. If the IPv4 IPFIX template is present and IPFIX flow records contain valid and expected field values, then Distributed Firewall IPFIX data is being generated and transmitted correctly from NSX to Aria Operations for Networks. If AON does not display the data despite correct IPFIX transmission, further investigation should focus on the AON ingestion and processing pipeline rather than NSX DFW.

 

Powered by Wolken Software