Edge NIC out of receive / transmit buffer alarm

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

Title: Alarm for Edge NIC out of receive buffer / Edge NIC out of transmit buffer
Event ID: edge_health.edge_nic_out_of_receive_buffer / edge_health.edge_nic_out_of_transmit_buffer
Alarm Description

• Purpose: Indicates overflow of Edge NIC receive / transmit buffer.
• Impact: Traffic drop will be observed. Rx / Tx misses / drop counter will keep increasing.

Environment

VMware NSX

Cause

NIC out of receive / transmit buffer alarm can be raised if the below conditions are observed in one or all CPUs.

• CPU usage is high, i.e. > 90%
• Rx / Tx pps is high
• Bursty traffic

Resolution

For NSX versions 3.x and above, follow the below steps:

Run the NSX CLI command 'get dataplane cpu stats' on the edge node and check:

• If cpu usage is high, i.e. > 90%, then take multiple samples of logical router interfaces stats using the command 'get logical-router interface stats' and if IPSec tunnel is enabled in the topology, then check IPsec tunnel stats using the command 'get ipsecvpn tunnel stats'. Then analyze the stats to see if majority of traffic is fragmented packets or IPSec packets. If yes, then it is expected behavior. If not, datapath is probably busy with other operations. If this alarm lasts more than 2-3 minutes, contact Broadcom Support.
• If cpu usage is not high, i.e., < 90%, then check if rx / tx pps is high using the command 'get dataplane cpu stats' (just to make sure the traffic rate is increasing). Double the ring size by using the command set dataplane ring-size rx <ring-size> or set dataplane ring-size tx <ring-size>. For example, if the current size is 512, increase it to 1024; if it is 1024, increase it to 2048 (The maximum value is 4096).
  • Changing the ring size requires a dataplane service restart 'restart service dataplane'. Note: Restarting dataplane service impacts datapath. To find the current ring buffer size set on the network adapter of the Edge node, refer: Finding Ring Buffer Size and MTU for Virtual Machine Network Interfaces
    Note: The continuous increase of ring size can lead to some performance issues. If even after increasing the ring size, the issue persists then it is an indication that edge needs a larger form factor deployment to accommodate the traffic. Refer NSX Edge VM System Requirements for more information on the system requirements for deploying a larger form factor edge.
• If the alarm keeps flapping i.e., triggers and resolves very soon, then it is due to bursty traffic. In this case check if rx / tx pps as described above, if it is not high during the alarm active period then contact Broadcom Support. If pps is high it confirms bursty traffic. Consider suppressing the alarm.
  Note: There is no specific benchmark to decide what is regarded as a high pps value. It depends on infrastructure and type of traffic. The comparison can be made by noting down when alarm is inactive and when it is active.

• The "edge_nic_out_of_transmit_buffer" error indicates that the ESX host is unable to process all the packets being transmitted by the Edge. Migrating some workloads to a different ESXi host, separate from the one hosting the Edge, can help alleviate the burst and improve performance.

Maintenance window required for remediation? Yes

Additional Information

Large number of "Receive packets dropped" & "Transmit packets dropped" on ESXi NIC's