TX traffic via LACP LAG bonding configured with bare metal NSX Edge is skewed to one interface.

Products

• VMware NSX

Issue/Introduction

• LACP LAG bonding is configured on bare metal NSX Edge nodes.
• The devices forming the bond span multiple NUMA nodes.
• Packets sent from LACP are skewed toward interfaces belonging to NUMA node 0.

Environment

VMware NSX 
VMware NSX-T Data Center 

Cause

This is expected behavior.
As explained in Install NSX Edge on Bare Metal, when devices forming the bond span multiple NUMA nodes, the bond sends packets using only the CPU of the network device on the local NUMA node 0.
Therefore, not all devices are used to balance traffic sent from the bonded devices.

 

Resolution

Configure LACP LAG bonding by specifying the interfaces belonging to NUMA node 0.

Additional Information

In this case, failover still works as failover is exclusive of load balancing.
If the Ethernet device attached to the local NUMA node is down, then the bond sends traffic to the other device even though it is not NUMA local.
The load-balancing optimization does not impact failover functionality.
Run get dataplane command to view the NUMA node associated with each datapath interface.
To move nics associated with datapath to single NUMA node, physical reconfiguration of the server is required via the BIOS.