Edge NIC out of receive / transmit buffer alarm

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

Title: Alarm for Edge NIC out of receive buffer / Edge NIC out of transmit buffer
Event ID: edge_health.edge_nic_out_of_receive_buffer / edge_health.edge_nic_out_of_transmit_buffer
Alarm Description

• Purpose: Indicates overflow of Edge NIC receive / transmit buffer.
• Impact: Traffic drop will be observed. Rx / Tx misses / drop counter will keep increasing.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

NIC out of receive / transmit buffer alarm can be raised if the below conditions are observed in one or all CPUs.

• CPU usage is high, i.e. > 90%
• Rx / Tx pps is high
• Bursty traffic

Resolution

Steps to resolve
For 3.0.0 and higher

Recommended Action:

Run the NSX CLI command 'get dataplane cpu stats' on the edge node and check:

• If cpu usage is high, i.e. > 90%, then take multiple samples of logical router interfaces stats using the command 'get logical-router interface stats' and if IPSec tunnel is enabled in the topology, then check IPsec tunnel stats using the command 'get ipsecvpn tunnel stats'. Then analyze the stats to see if majority of traffic is fragmented packets or IPSec packets. If yes, then it is expected behavior. If not, datapath is probably busy with other operations. If this alarm lasts more than 2-3 minutes, contact Broadcom Support.
• If cpu usage is not high, i.e., < 90%, then check if rx / tx pps is high using the command 'get dataplane cpu stats' (just to make sure the traffic rate is increasing). Then increase the ring size by 1024 using the command 'set dataplane ring-size rx <ring-size>' or 'set dataplane ring-size tx <ring-size>'.
  Note: The continuous increase of ring size by 1024 factor can lead to some performance issues. If even after increasing the ring size, the issue persists then it is an indication that edge needs a larger form factor deployment to accommodate the traffic.
• If the alarm keeps on flapping i.e., triggers and resolves very soon, then it is due to bursty traffic. In this case check if rx / tx pps as described above, if it is not high during the alarm active period then contact Broadcom Support. If pps is high it confirms bursty traffic. Consider suppressing the alarm.
  Note: There is no specific benchmark to decide what is regarded as a high pps value. It depends on infrastructure and type of traffic. The comparison can be made by noting down when alarm is inactive and when it is active.
• The "edge_nic_out_of_transmit_buffer" error indicates that the ESX host is unable to process all the packets being transmitted by the Edge. Migrating some workloads to a different ESXi host, separate from the one hosting the Edge, can help alleviate the burst and improve performance.

Maintenance window required for remediation? Yes