Edge NIC Out Of Transmit Buffer / Edge NIC Transmit Queue Overflow alarms are being triggered constantly.

Products

• VMware NSX

Issue/Introduction

• "vmxnet3_process_events(): queue error event" and "vmxnet3_process_events(): tq 1 error 0x80000007 (or tq 0)" errors are continuously getting logged in the /var/log/syslog of the NSX Edge VM.
• A recurrent "Edge NIC Out Of Transmit Buffer" alarm might also be observed on the NSX Manager.
• Intermittent Packet drops on the Edges are seen.
• ENS Is enabled on the ESXi Host where the Edge VM is present.
• Default Transport VLAN 0 is configured on the Uplink Profile (Overlay Edge Switch) associated with the Edge Node VM.
  
  
• Below Error message are continuously seen on the Edge /var/log/sylog in addition to the "edge_nic_out_of_transmit_buffer" and "edge_nic_transmit_queue_overflow"
  YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] vmxnet3_process_events(): queue error event 0x2 for 00:50:56:##:##:##
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] vmxnet3_process_events(): tq 1 error 0x80000007
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="phys-port" level="INFO"] reset dpdk callback for port 3
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] Posted mgmt lsn message mgmt: port 3 type DPDK_RESET
.....
.....
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] vmxnet3_process_events(): queue error event 0x2 for 00:50:56:##:##:##
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="WARN"] vmxnet3_process_events(): tq 0 error 0x80000007
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="phys-port" level="INFO"] reset dpdk callback for port 3
YYYY-MM-DDTHH:MM:SS:sssZ <edge_name> NSX 2469204 FABRIC [nsx@6876 comp="nsx-edge" subcomp="datapathd" s2comp="intel-rte" level="INFO"] Posted mgmt lsn message mgmt: port 3 type DPDK_RESET

Note: The preceding log excerpts are only examples. Date, time, port and queue numbers as well as other variables may vary depending on your environment.

Environment

VMware NSX 

Cause

The issue is caused by the edge attempting to send a packet on an uplink with VLAN 0 and the Tx VLAN offload bit set in the packet metadata. The ESXi's ENS interface flags this as a fatal error, drops the packet, and causes the edge dataplane to reset the uplink device due to a "VMXNET3_ERR_TXD_INVALID (0x80000007)" error due to an invalid TX descriptor

Resolution

Workaround

• In the NSX Manager Change the Transport VLAN of the Edge Uplink profile (Overlay Edge Switch) associated with the affected Edge nodes from the Default Transport VLAN 0 to the explicitly required environment VLAN.

• Kindly make sure the Uplink Mapping associated with NSX Edge VM is configured as Trunk VLAN port with required VLAN allowed instead of Access VLAN Port

 

 

Additional Information

KB for Edge NIC out of receive / transmit buffer alarm