Asymmetric Inbound Traffic Loss on ESXi Hosts Due to Cisco Nexus TTAG

Products

• VMware vSphere ESXi
• VMware vCenter Server
• VMware NSX

Issue/Introduction

In Cisco Nexus switching fabric environments, virtual machines or VMkernel (vmk) interfaces may experience a total loss of inbound (return) network traffic while outbound traffic remains functional. Specifically, downlinks connecting to VMware ESXi hypervisor hosts experience selective or complete packet loss on incoming routed traffic paths when Cisco Timestamp Tagging (ttag) is enabled on the physical switch.

• Affected Virtual Machines and VMkernel interfaces retain outbound connectivity but experience a 100% loss of inbound (return) connectivity for one or more switch-local Layer 2/Layer 3 routed paths, causing an asymmetric traffic blackhole.

• Both the impacted ESXi host and the target IP can successfully ping the gateway IP of the affected vm/vmk VLAN, confirming the routing table is populated correctly.

• The standard packet tracing utility pktcap-uw, when configured with L2–L4 filters (e.g., pktcap-uw --trace --uplink <vmnic#> --ip <target_IP>), returns completely blank results and fails to capture any return frames.

• Host physical NIC statistics (extracted via nicinfo.sh or esxcli network nic stats) report a completely clean baseline (zero Receive Packets Dropped, Missed Errors, Protocol Anomalies, or CRC/FCS Errors).

     NIC statistics for vmnic#:
        Packets received: 30372801
        Bytes received: 91824317628
        Receive packets dropped: 0       <<<<<<<<<<
        Multicast packets received: 17451
        Broadcast packets received: 1323
        Total receive errors: 0
        Receive length errors: 0
        Receive over errors: 0
        Receive CRC errors: 0
        Receive frame errors: 0
        Receive FIFO errors: 0
        Receive missed errors: 0

• Hardware packet captures executed by the network team on the Cisco switch (via ELAM or SPAN) confirm the switch is successfully transmitting return packets out of the physical downlink interface facing the ESXi host with zero drops.

• Network firewalls and security policies along the path have been verified, confirming no explicit deny rules are blocking the return traffic between the source and destination.

• Executing vsish -e get /net/portsets/<DvsPortset-#>/ports/<port_id>/stats (vDS) or vsish -e get /net/portsets/<vSwitch#>/ports/<port_id>/stats (vSS) uplink port shows active, incrementing counts on the droppedExceptionRx counter for the affected physical uplink. (Note: Run net-stats -l | grep -i vmnic to map runtime port IDs).
  

  /> get  /net/portsets/<DvsPortset-#>/ports/<port_id>/stats 
  packet stats {
     pktsTx:7581450260
     pktsTxMulticast:4606665
     pktsTxBroadcast:46954
     pktsRx:10832574821
     pktsRxMulticast:113129557
     pktsRxBroadcast:280724
     droppedTx:0
     droppedRx:9234                  <<<<<<<<<<<
     droppedClientTx:0
     droppedIOchainTx:0
     droppedExceptionTx:0
     droppedEnsTx:0
     droppedOtherTx:0
     droppedClientRx:0
     droppedIOchainRx:0
     droppedExceptionRx:9234         <<<<<<<<<<<
     droppedEnsRx:0
     droppedOtherRx:0
  }
  

   

• The Nexus switch is configured with feature ptp in the global configuration and ttag on the physical switch interface/port-channel facing the <target_IP>.

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

This is expected behavior when a virtual switch receives an unsupported, proprietary protocol layout. The root cause is a protocol parsing failure at the ESXi kernel switch layer, driven by Cisco Timestamp Tagging (TTAG) metadata leaking onto the host uplink.

Cisco TTAG (EtherType 0x8905) is a proprietary Layer 2 tracking tag utilized for hardware-level telemetry on Nexus CloudScale platforms. It operates entirely separately from the open-standard IEEE 1588 PTP over Ethernet (0x88F7), which ESXi natively parses and supports.

When a Nexus switch generates TTAG data, the egress ASIC inserts a custom Layer 2 shim header. This encapsulation alters the outer Ethernet frame, setting the primary EtherType to 0x8905. Because this proprietary header shifts the expected payload structure, it masks the inner IPv4 (0x0800) or IPv6 (0x86DD) EtherTypes and alters the expected protocol and Next Header byte offsets.

The physical ESXi NIC receives the frame, validates the switch-recalculated CRC, and successfully executes a Direct Memory Access (DMA) transfer into host memory, registering zero physical Rx errors. However, because the ESXi virtual switch lacks a registered protocol handler for the proprietary 0x8905 EtherType, the parser aborts processing, drops the frame, and increments the droppedExceptionRx counter.

Resolution

This issue must be permanently remediated at the physical switch egress layer, as ESXi cannot be configured to recognize vendor-locked proprietary L2 framing.

• Engage your physical network team or switch vendor to correct the physical switch configuration on the downstream switchports connecting to the ESXi hosts.

• Instruct the switch ASIC to shear the proprietary 0x8905 tracking header from the payload right before serialization onto the physical wire by applying the appropriate stripping command to the egress interface.

Note: Broadcom does not provide or maintain configuration instructions for third-party hardware. The following Cisco NX-OS commands are provided as an example only. You must validate the exact syntax with your network vendor or the official "Cisco NX-OS System Management Configuration Guide" applicable to your specific switch model and OS version.

Example NX-OS Configuration:

switch(config)# interface <interface_id>
switch(config-if)# ttag            <-------- Adds timestamp tagging on the ingress interface
switch(config-if)# ttag-strip      <-------- Removes TTAG from egress packets on the interface (REQUIRED FOR ESXi)

Additional Information

Troubleshooting virtual machine network connection issues.

Packet capture on ESXi using the pktcap-uw tool.

To confirm whether the physical fabric is polluting the ingress path with TTAG frames, you can bypass standard IP/MAC filters and explicitly capture the proprietary EtherType from the ESXi host.

• • Execute the following kernel trace: pktcap-uw --uplink vmnic# --ethtype 0x8905 --trace 

Third-Party Documentation and Architectural References:

• Cisco Configuration Guide: Official documentation explicitly confirms that if TTAG is attached to non-Nexus devices that do not support EtherType 0x8905, stripping the tag on egress is required; otherwise, the non-Nexus devices will drop the packet. Configuring PTP - Cisco Nexus 9000 Series NX-OS System Management Configuration Guide

• Tag Switching Architecture (RFC 2105): For a deeper understanding of why shim headers mask inner payloads from non-participating edge devices, refer to the foundational IETF documentation on Cisco Tag Switching. RFC 2105: Cisco Systems' Tag Switching Architecture Overview

Disclaimer: Broadcom is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that Broadcom endorses, recommends, or accepts any responsibility for the content of such sites.

For further assistance Contact Broadcom support.