Buffer Overrun issues with NSX Edge during large file transfer

Products

• VMware NSX

Issue/Introduction

• Observing the following alarm in the NSX UI: Edge NIC out of receive / transmit buffer alarm

  • Edge NIC out of receive buffer

  • Edge NIC out of transmit buffer

• After or during a large file transfer flowing through NSX Edge Node data path, the above alarm shows

• NSX Edge Nodes are Large form factor size or smaller

• Similar NSX Edge Node logging is observed: Edge NIC fp-eth0 receive ring buffer has overflowed by 6.785233% on Edge node ########-####-####-####-############. The missed packet count is 2270695 and processed packet count is 31194550.

• Tier1 gateways deployed by Tanzu/VKS are all active on a single NSX Edge Node

Environment

VMware NSX

Cause

Multiple factors in this scenario fall under the cause:

• Rx Errors observed on Edge nodes - This was observed on the NSX Edge Node and ESXi host where the NSX Edge Node in question lives

• Resize NSX Edge Node - Large form factor or smaller has a ring size of 2048 or lower

• Load balance/rebalance existing T1 gateways in the edge cluster in NSX environment - Having all active Tier1's living on the same NSX Edge Node will cause overlay traffic from all ESXi hosts in the Overlay Transport Zone to send traffic to the one NSX Edge Node housing all active Tier1 routers

Resolution

Follow the resolution steps from the following three KBs:

• Rx Errors observed on Edge nodes - The resolution steps will allow an increase to the queue size limit for handling bursting traffic

• Resize NSX Edge Node - Xlarge form factor will give more resources to the NSX Edge Node along with a ring buffer size of 4096

• Load balance/rebalance existing T1 gateways in the edge cluster in NSX environment - This process will balance the Tier1 logical routers between the NSX Edge Nodes in the Edge Cluster to allow for a more even traffic distribution across the Edge Nodes